Upload
alexandru-ionut-pohontu
View
243
Download
0
Embed Size (px)
Citation preview
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 1110
Testing Course
explaining the
ISTQB Certified Tester Foundation Level Syllabus
1
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 2110
111 Why is testing necessary ndash Why do we test(hellipthe common answer is ) To find bugs
but consider alsobullTo reduce the impact of the failures at the clientrsquos site (live defects) andensure that they will not affect costs amp profitabilitybullTo decrease the rate of failures (increase the productrsquos reliability)
bullTo improve the quality of the productbullTo ensure requirements are implemented fully amp correctlybullTo validate that the product is fit for its intended purposebullTo verify that required standards and legal requirements are metbullTo maintain the company reputation
Testing provides the productrsquos measure of quality
Can we test everything Exhaustive testing is possible
-No sorry helliptime amp resources make it impractical hellipbut instead-We must understand the risk to the clientrsquos business of the software notfunctioning correctlybullWe must manage and reduce risk carry out a Risk Analysis of the
applicationbullPrioritise tests to focus them (time amp resources) on the main areas of risk2
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 3110
111 Why is testing necessary - Testing goals and objectives
The main goals of testing
bull Find defectsbull Assess the level of quality of the software product and providing relatedinformation to the stakeholdersbull Prevent defectsbull Reduce risk of operational incidents
bull Increase the product qualityDifferent viewpoints and objectives
bull Unit amp integration test ndash find as many defects as possiblebull Acceptance testing ndash confirm that the system works as specified and that thequality is good enoughbull Testing metrics gathering ndash provide information to the project manager aboutthe product quality and the risks involved
bull Design tests early and review requirements ndash help prevent defects3
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 4110
112 Why is testing necessary - Testing Glossary
A programmer (or analyst) can make an error (mistake) which produces a defect(fault bug) in the programrsquos code If such a defect in the code is executed thesystem will fail to do what it should do (or it will do something it should not do)causing a failure
Error (mistake) = a human action that produces an incorrect resultDefect (bug) = a flaw that can cause the component or system to fail to perform its
required functionFailure = deviation of the component or system from its expected delivery serviceor result
Anomaly = any condition that deviates from expectations based on requirementsspecifications design documents user documentation standards or someonersquosperceptions or expectations
Defect masking = An occurrence in which one defect prevents the detection ofanother
4
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 5110
112 Why is testing necessary ndash Causes of the errorsDefects are caused by human errors
Why Because ofTime pressure - the more pressure we are under the more likely we are tomake mistakesbullCode complexity or new technology
bullToo many system interactionsbullRequirements not clearly defined changed amp not properly documentedbullWe make wrong assumptions about missing bits of informationbullPoor communicationbullPoor training
5
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 6110
112 Why is testing necessary - Causes of software defects -Defects taxonomy
6
(Boris Beizer)
bull Requirements (incorrect logic completeness verifiability documentation changes)bull Features and functionality (correctness missing case domain and boundarymessages exception mishandled)
bull Structural (control flow sequence data processing)bull Data (definition structure access handling)bull Implementation and Codingbull Integration (internal and external interfaces)bull System (architecture performance recovery partitioning environment)bull Test definition and execution (test design test execution documentation reporting)
(Cem Kaner (b1))bull User interface (functionality communication missing performance output)bull Error handling (prevention detection recovery)bull Boundary (numeric loops)
bull Calculation (wrong constants wrong operation order over amp underflow)bull Initialization (data item string loop control)bull Control flow (stop crash loop if-then-elsehellip)bull Data handling (data type parameter list values)bull Race amp Load conditions (event sequence no resources)bull Source and version control (old bug reappear)bull Testing (fail to notice fail to test fail to report)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 7110
113 Why is testing necessary - The role of testing in thesoftware life cycle
Testers do cooperate with Analysts ndash to review the specifications for completeness and correctnessensure that they are testablebull Designers ndash to improve interfaces testability and usabilitybull Programmers ndash to review the code and assess structural flawsbull Project manager ndash to estimate plan develop test cases perform tests andreport bugs to assess the quality and risksbull Quality assurance staff ndash to provide defect metricsbull Interactions with these project roles are very complex
RACI matrix (Responsible Accountable Consulted Informed)
7
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 8110
114 Why is testing necessary ndash What is quality
8
Quality (ISO) = The totality of the characteristics of an entity that bear on its
ability to satisfy stated or implied needs
bull there are many more definitions
Testing means not only to Verify (the thing is done right)hellip but also to Validate (the right thing is done)
Software quality includes reliability usability efficiency maintainability andportabilityRELIABILITY The ability of the software product to perform its requiredfunctions under stated conditions for a specified period of time or for aspecified number of operationsUSABILITY The capability of the software to be understood learned used
and attractive to the user when used under specified conditionsEFFICIENCY The capability of the software product to provide appropriateperformance relative to the amount of resources used under stated conditionsMAINTAINABILITY The ease with which a software product can be modifiedto correct defects modified to meet new requirements modified to make futuremaintenance easier or adapted to a changed environmentPORTABILITY The ease with which the software product can be transferredfrom one hardware or software environment to another
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 9110
114 Why is testing necessary - Testing and quality
Testing does not inject Quality
into the product but will measurethe level of the Quality of theproduct
Quality can be measured forbull progressbull variance (planned versus actual)
Measuring product quality bullFunctional compliance - functional software requirements testing
bullNon functional requirementsbullTest coverage criteriabullDefect count or defect trend criteria
9
1 1 4 Wh i i li ib
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 10110
114 Why is testing necessary ndash quality attributes
The QUINT model (extended ISO model)
10
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 11110
115 Why is testing necessary - How much testing is enough
when to stop testingThe five basic criteria often used to decide are
bull Previously defined coverage goals have been metbull The defect discovery rate has dropped below a previously defined thresholdbull The cost of finding the next defect exceeds the expected loss from that defectbull The project team reaches consensus that it is appropriate to release the productbull The manager decides to deliver the product
All these criteria are risk basedIt is important not depending on only one stopping criterion
Software Reliability Engineering can help also to determine when to stop testingby taking into consideration aspects like failure intensity
11
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 12110
12 What is testing - Defini tion of testing
12
Testing = the process concerned with planning the necessary static and dynamic
activities preparation and evaluation of software products and related deliverablesin order tobull determine that they satisfy specified requirementsbull demonstrate that they are fit for the intended usebull detect defects help and motivate the developers to fix thembull measure assess and improve the quality of the software product
Testing should be performed throughout the whole software life cycle
There are two basic types of testing execution and non-execution based
Other definitions
bull(IEEE) Testing = the process of analyzing a software item to detect the differencesbetween existing and required conditions and to evaluate its featuresbull(Myers (b3)) Testing = the process of executing a program with the intent of findingerrorsbull(Craig amp Jaskiel (b5)) Testing = a concurrent lifecycle process of engineering usingand maintaining test-ware in order to measure and improve the quality of thesoftware being tested
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 13110
12 What is testing ndash Testing ldquo schoolsrdquo
13
Analytic School - testing is rigorous academic and technicalbullTesting is a branch of CSMathematicsbullTesting techniques must have a logic-mathematical formbullKey Question Which techniques should we usebullRequire precise and detailed specifications
Factory School - testing is a way to measure progress with emphasis on cost and
repeatable standardsbullTesting must be managed amp cost effectivebullTesting validates the product amp measures development progressbullKey Questions How can we measure whether wersquore making progress When will we be donebullRequire clear boundaries between testing and other activities (startstop criteria)
bullEncourage standards (V-model) ldquobest practicesrdquo and certificationQuality School - emphasizes process amp quality acting as the gatekeeper bullSoftware quality requires disciplinebullTesters may need to police developers to follow the rulesbullKey Question Are we following a good process
bullTesting is a stepping stone to ldquoprocess improvementrdquoContext-Driven School - emphasizes people setting out to find the bugs that will bemost important to stakeholders
bullSoftware is created by people People set the contextbullTesting finds bugs acting as a skilled mental activitybullKey Question What testing would be most valuable right nowbullExpect changes Adapt testing plans based on test resultsbullTesting research requires empirical and psychological study
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 14110
13 General testing principles
14
bull Testing shows presence of defects but cannot prove that there are no moredefects testing can only reduce the probability of undiscovered defects
bull Complete exhaustive testing is impossible good strategy and risk managementmust be used
bull Pareto rule (defect clustering) usually 20 of the modules contain 80 of thebugs
bull Early testing testing activities should start as soon as possible (including hereplanning design reviews)
bull Pesticide paradox if the same set of tests are repeated over again no new bugswill be found the test cases should be reviewed modified and new test casesdeveloped
bull Context dependence test design and execution is context dependent (desktopweb applications real-time hellip)
bull Verification and Validation discovering defects cannot help a product that is not fitto the users needs
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 15110
13 General testing principles ndash heuristics of software testing
bullOperability - The better it works the more efficiently it can be tested
bullObservability - What we see is what we test
bull Controllability - The better we control the software the more the testing processcan be automated and optimized
bull Decomposability - By controlling the scope of testing we can quickly isolateproblems and perform effective and efficient testing
bull Simplicity - The less there is to test the more quickly we can test it
bull Stability - The fewer the changes the fewer are the disruptions to testing
bull Understandability - The more information we will have the smarter we will test
bull Suitability - The more we know about the intended use of the software the
better we can organize our testing to find important bugs15
1 4 1 F d t l t t h
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 16110
141 Fundamental test process ndash phases
-Test Planning amp Test control
-Test Analysis amp Design
-Test Implementation ampExecution
-Evaluating exit criteria ampReporting
-Test Closure activities
16
1 4 1 Fundamental test process planning amp control
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 17110
141 Fundamental test process ndash planning amp control
Planning1 Determine scope
bull Study project documents used software life-cycle specifications product desiredquality attributesbull Clarify test process expectations
2 Determine risksbull Choose quality risk analysis method (eg FMEA)
bull Document the list of risks probability impact priority identify mitigation actions3 Estimate testing effort determine costs develop schedule
bull Define necessary rolesbull Decompose test project into phases and tasks (WBS)bull Schedule tasks assign resources set-up dependencies
4 Refine planbull Select test strategy (how to do it what test types at which test levels)bull Select metrics to be used for defect tracking coverage monitoringbull Define entry and exit criteria
Controlbull Measure and analyze resultsbull Monitor testing progress coverage exit criteriabull Assign or reallocate resources update the test plan schedulebull Initiate corrective actionsbull Make decisions
17
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 18110
142 Fundamental test process ndash analysis amp design
bull Reviewing the test basis (such as requirements architecture designinterfaces)
bull Identifying test conditions or test requirements and required test data
based on analysis of test items and the specificationbull Designing the testsbull Choose test techniquesbull Identify test scenarios pre-conditions expected results post-
conditionsbull Identify possible test Oracles
bull Evaluating testability of the requirements and systembull Designing the test environment set-up and identifying any required
infrastructure and tools
(see Lee Copeland (b2))
18
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 19110
142 Fundamental test process ndash what is a test Oracle
19
bull The expected result (test outcome) must be defined at the test analysis stagebull Who will decide that (expected result = actual result) when the test will be
executed The Test Oracle
Test Oracle = A source to determine expected result a principle or mechanism torecognize a problem The Test Oracle can bebull an existing system (the old versionhellip)bull a document (specification user manual)bull a competent client representative
hellip but never the source code itself Oracles in use = Simplification of Risk do not assess lsquopass - failrsquo but instead
lsquoproblem - no problemrsquo
Problem Oracles and Automation - Our ability to automate testing isfundamentally constrained by our ability to create and use oracles
Possible issues
bull false alarmsbull missed bugs
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 20110
143 Fundamental test process ndash implementation amp execution
Test implementation
bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)
bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission
(see Rex Black (b4))
20
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 21110
143 Fundamental test process ndash priori tizing the Test Cases
Why prioritize the Test Cases
bullIt is not possible to test everything we must do our best in the time available
bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence
bullThis means we must prioritise and focus testing on the priorities
What to watch
bullSeverity of possible defectsbullProbability of possible defects
bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity
21
144 Fundamental test process ndash evaluating exit criteria and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 22110
144 Fundamental test process evaluating exit criteria andreporting
Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed
bull Check if testing mission should be changed
Test report ing
bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity
(see Rex Black (b4) amp RUP- Test discipline(s5))
22
1 4 5 F d l l
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 23110
145 Fundamental test process ndash test closure
bull Verify if test deliverables have been delivered
bull Check and close the remaining active bug reports
bull Archiving the test-ware and environment
bull Handover of the test environment
bull Analyze the identified test process problems (lessons learned)
bull Implement action plan based improvements
(see Rex Black (b4))
23
1 5 Th h l f i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 24110
15 The psychology of testing
24
Testing is regarded as a lsquodestructiversquo activity
(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts
bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise
bullShould be a planned organised kind ofperson
Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices
Rex BlackrsquosTop 10 professional errors
bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations
(see also Brian Marickrsquos article )
1 5 Th h l g f t ti g
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 25110
15 The psychology of testing
25
ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)
ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects
bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs
Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)
Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts
bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence
2 1 1 The V testing model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 26110
211 The V testing model
26
211 The V testing model ndash Verification amp Validationifi i C fi i b
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 27110
27
Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled
Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled
Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level
2 1 1 The W testing model ndash dynamic testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 28110
211 The W testing model ndash dynamic testing
28
2 1 1 The W testing model ndash static testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 29110
211 The W testing model static testing
29
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 30110
212 Software development models Waterfall
30
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 31110
212 Software development models Waterfall
Waterfall weaknesses
middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule
middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover
middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen
middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to
partition the system for delivery of pieces of the system
31
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 32110
p p yp
32
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 33110
p p yp
Rapid Prototype Model weaknesses
middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked
middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products
middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations
middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary
middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study
33
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 34110
34
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 35110
35
Incremental Model weaknesses
middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments
middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-
defined interfaces are required
middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 36110
36
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 37110
Spiral Model weaknesses
middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to
proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk
analysis and prototyping may be excessive
37
212 Software development models - Rational Unified Process
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 38110
38
213 Software development models ndash Testing life cycle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 39110
bull For each software activity there must be a corresponding testing activity
bull The objectives of the testing are specific to that ldquotestedrdquo activity
bull Plan analysis and design of a testing activity should be done during thecorresponding development activity
bull Review and inspection must be considered as part of testing activities
39
221 Test levels ndash Component testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 2110
111 Why is testing necessary ndash Why do we test(hellipthe common answer is ) To find bugs
but consider alsobullTo reduce the impact of the failures at the clientrsquos site (live defects) andensure that they will not affect costs amp profitabilitybullTo decrease the rate of failures (increase the productrsquos reliability)
bullTo improve the quality of the productbullTo ensure requirements are implemented fully amp correctlybullTo validate that the product is fit for its intended purposebullTo verify that required standards and legal requirements are metbullTo maintain the company reputation
Testing provides the productrsquos measure of quality
Can we test everything Exhaustive testing is possible
-No sorry helliptime amp resources make it impractical hellipbut instead-We must understand the risk to the clientrsquos business of the software notfunctioning correctlybullWe must manage and reduce risk carry out a Risk Analysis of the
applicationbullPrioritise tests to focus them (time amp resources) on the main areas of risk2
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 3110
111 Why is testing necessary - Testing goals and objectives
The main goals of testing
bull Find defectsbull Assess the level of quality of the software product and providing relatedinformation to the stakeholdersbull Prevent defectsbull Reduce risk of operational incidents
bull Increase the product qualityDifferent viewpoints and objectives
bull Unit amp integration test ndash find as many defects as possiblebull Acceptance testing ndash confirm that the system works as specified and that thequality is good enoughbull Testing metrics gathering ndash provide information to the project manager aboutthe product quality and the risks involved
bull Design tests early and review requirements ndash help prevent defects3
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 4110
112 Why is testing necessary - Testing Glossary
A programmer (or analyst) can make an error (mistake) which produces a defect(fault bug) in the programrsquos code If such a defect in the code is executed thesystem will fail to do what it should do (or it will do something it should not do)causing a failure
Error (mistake) = a human action that produces an incorrect resultDefect (bug) = a flaw that can cause the component or system to fail to perform its
required functionFailure = deviation of the component or system from its expected delivery serviceor result
Anomaly = any condition that deviates from expectations based on requirementsspecifications design documents user documentation standards or someonersquosperceptions or expectations
Defect masking = An occurrence in which one defect prevents the detection ofanother
4
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 5110
112 Why is testing necessary ndash Causes of the errorsDefects are caused by human errors
Why Because ofTime pressure - the more pressure we are under the more likely we are tomake mistakesbullCode complexity or new technology
bullToo many system interactionsbullRequirements not clearly defined changed amp not properly documentedbullWe make wrong assumptions about missing bits of informationbullPoor communicationbullPoor training
5
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 6110
112 Why is testing necessary - Causes of software defects -Defects taxonomy
6
(Boris Beizer)
bull Requirements (incorrect logic completeness verifiability documentation changes)bull Features and functionality (correctness missing case domain and boundarymessages exception mishandled)
bull Structural (control flow sequence data processing)bull Data (definition structure access handling)bull Implementation and Codingbull Integration (internal and external interfaces)bull System (architecture performance recovery partitioning environment)bull Test definition and execution (test design test execution documentation reporting)
(Cem Kaner (b1))bull User interface (functionality communication missing performance output)bull Error handling (prevention detection recovery)bull Boundary (numeric loops)
bull Calculation (wrong constants wrong operation order over amp underflow)bull Initialization (data item string loop control)bull Control flow (stop crash loop if-then-elsehellip)bull Data handling (data type parameter list values)bull Race amp Load conditions (event sequence no resources)bull Source and version control (old bug reappear)bull Testing (fail to notice fail to test fail to report)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 7110
113 Why is testing necessary - The role of testing in thesoftware life cycle
Testers do cooperate with Analysts ndash to review the specifications for completeness and correctnessensure that they are testablebull Designers ndash to improve interfaces testability and usabilitybull Programmers ndash to review the code and assess structural flawsbull Project manager ndash to estimate plan develop test cases perform tests andreport bugs to assess the quality and risksbull Quality assurance staff ndash to provide defect metricsbull Interactions with these project roles are very complex
RACI matrix (Responsible Accountable Consulted Informed)
7
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 8110
114 Why is testing necessary ndash What is quality
8
Quality (ISO) = The totality of the characteristics of an entity that bear on its
ability to satisfy stated or implied needs
bull there are many more definitions
Testing means not only to Verify (the thing is done right)hellip but also to Validate (the right thing is done)
Software quality includes reliability usability efficiency maintainability andportabilityRELIABILITY The ability of the software product to perform its requiredfunctions under stated conditions for a specified period of time or for aspecified number of operationsUSABILITY The capability of the software to be understood learned used
and attractive to the user when used under specified conditionsEFFICIENCY The capability of the software product to provide appropriateperformance relative to the amount of resources used under stated conditionsMAINTAINABILITY The ease with which a software product can be modifiedto correct defects modified to meet new requirements modified to make futuremaintenance easier or adapted to a changed environmentPORTABILITY The ease with which the software product can be transferredfrom one hardware or software environment to another
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 9110
114 Why is testing necessary - Testing and quality
Testing does not inject Quality
into the product but will measurethe level of the Quality of theproduct
Quality can be measured forbull progressbull variance (planned versus actual)
Measuring product quality bullFunctional compliance - functional software requirements testing
bullNon functional requirementsbullTest coverage criteriabullDefect count or defect trend criteria
9
1 1 4 Wh i i li ib
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 10110
114 Why is testing necessary ndash quality attributes
The QUINT model (extended ISO model)
10
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 11110
115 Why is testing necessary - How much testing is enough
when to stop testingThe five basic criteria often used to decide are
bull Previously defined coverage goals have been metbull The defect discovery rate has dropped below a previously defined thresholdbull The cost of finding the next defect exceeds the expected loss from that defectbull The project team reaches consensus that it is appropriate to release the productbull The manager decides to deliver the product
All these criteria are risk basedIt is important not depending on only one stopping criterion
Software Reliability Engineering can help also to determine when to stop testingby taking into consideration aspects like failure intensity
11
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 12110
12 What is testing - Defini tion of testing
12
Testing = the process concerned with planning the necessary static and dynamic
activities preparation and evaluation of software products and related deliverablesin order tobull determine that they satisfy specified requirementsbull demonstrate that they are fit for the intended usebull detect defects help and motivate the developers to fix thembull measure assess and improve the quality of the software product
Testing should be performed throughout the whole software life cycle
There are two basic types of testing execution and non-execution based
Other definitions
bull(IEEE) Testing = the process of analyzing a software item to detect the differencesbetween existing and required conditions and to evaluate its featuresbull(Myers (b3)) Testing = the process of executing a program with the intent of findingerrorsbull(Craig amp Jaskiel (b5)) Testing = a concurrent lifecycle process of engineering usingand maintaining test-ware in order to measure and improve the quality of thesoftware being tested
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 13110
12 What is testing ndash Testing ldquo schoolsrdquo
13
Analytic School - testing is rigorous academic and technicalbullTesting is a branch of CSMathematicsbullTesting techniques must have a logic-mathematical formbullKey Question Which techniques should we usebullRequire precise and detailed specifications
Factory School - testing is a way to measure progress with emphasis on cost and
repeatable standardsbullTesting must be managed amp cost effectivebullTesting validates the product amp measures development progressbullKey Questions How can we measure whether wersquore making progress When will we be donebullRequire clear boundaries between testing and other activities (startstop criteria)
bullEncourage standards (V-model) ldquobest practicesrdquo and certificationQuality School - emphasizes process amp quality acting as the gatekeeper bullSoftware quality requires disciplinebullTesters may need to police developers to follow the rulesbullKey Question Are we following a good process
bullTesting is a stepping stone to ldquoprocess improvementrdquoContext-Driven School - emphasizes people setting out to find the bugs that will bemost important to stakeholders
bullSoftware is created by people People set the contextbullTesting finds bugs acting as a skilled mental activitybullKey Question What testing would be most valuable right nowbullExpect changes Adapt testing plans based on test resultsbullTesting research requires empirical and psychological study
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 14110
13 General testing principles
14
bull Testing shows presence of defects but cannot prove that there are no moredefects testing can only reduce the probability of undiscovered defects
bull Complete exhaustive testing is impossible good strategy and risk managementmust be used
bull Pareto rule (defect clustering) usually 20 of the modules contain 80 of thebugs
bull Early testing testing activities should start as soon as possible (including hereplanning design reviews)
bull Pesticide paradox if the same set of tests are repeated over again no new bugswill be found the test cases should be reviewed modified and new test casesdeveloped
bull Context dependence test design and execution is context dependent (desktopweb applications real-time hellip)
bull Verification and Validation discovering defects cannot help a product that is not fitto the users needs
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 15110
13 General testing principles ndash heuristics of software testing
bullOperability - The better it works the more efficiently it can be tested
bullObservability - What we see is what we test
bull Controllability - The better we control the software the more the testing processcan be automated and optimized
bull Decomposability - By controlling the scope of testing we can quickly isolateproblems and perform effective and efficient testing
bull Simplicity - The less there is to test the more quickly we can test it
bull Stability - The fewer the changes the fewer are the disruptions to testing
bull Understandability - The more information we will have the smarter we will test
bull Suitability - The more we know about the intended use of the software the
better we can organize our testing to find important bugs15
1 4 1 F d t l t t h
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 16110
141 Fundamental test process ndash phases
-Test Planning amp Test control
-Test Analysis amp Design
-Test Implementation ampExecution
-Evaluating exit criteria ampReporting
-Test Closure activities
16
1 4 1 Fundamental test process planning amp control
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 17110
141 Fundamental test process ndash planning amp control
Planning1 Determine scope
bull Study project documents used software life-cycle specifications product desiredquality attributesbull Clarify test process expectations
2 Determine risksbull Choose quality risk analysis method (eg FMEA)
bull Document the list of risks probability impact priority identify mitigation actions3 Estimate testing effort determine costs develop schedule
bull Define necessary rolesbull Decompose test project into phases and tasks (WBS)bull Schedule tasks assign resources set-up dependencies
4 Refine planbull Select test strategy (how to do it what test types at which test levels)bull Select metrics to be used for defect tracking coverage monitoringbull Define entry and exit criteria
Controlbull Measure and analyze resultsbull Monitor testing progress coverage exit criteriabull Assign or reallocate resources update the test plan schedulebull Initiate corrective actionsbull Make decisions
17
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 18110
142 Fundamental test process ndash analysis amp design
bull Reviewing the test basis (such as requirements architecture designinterfaces)
bull Identifying test conditions or test requirements and required test data
based on analysis of test items and the specificationbull Designing the testsbull Choose test techniquesbull Identify test scenarios pre-conditions expected results post-
conditionsbull Identify possible test Oracles
bull Evaluating testability of the requirements and systembull Designing the test environment set-up and identifying any required
infrastructure and tools
(see Lee Copeland (b2))
18
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 19110
142 Fundamental test process ndash what is a test Oracle
19
bull The expected result (test outcome) must be defined at the test analysis stagebull Who will decide that (expected result = actual result) when the test will be
executed The Test Oracle
Test Oracle = A source to determine expected result a principle or mechanism torecognize a problem The Test Oracle can bebull an existing system (the old versionhellip)bull a document (specification user manual)bull a competent client representative
hellip but never the source code itself Oracles in use = Simplification of Risk do not assess lsquopass - failrsquo but instead
lsquoproblem - no problemrsquo
Problem Oracles and Automation - Our ability to automate testing isfundamentally constrained by our ability to create and use oracles
Possible issues
bull false alarmsbull missed bugs
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 20110
143 Fundamental test process ndash implementation amp execution
Test implementation
bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)
bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission
(see Rex Black (b4))
20
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 21110
143 Fundamental test process ndash priori tizing the Test Cases
Why prioritize the Test Cases
bullIt is not possible to test everything we must do our best in the time available
bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence
bullThis means we must prioritise and focus testing on the priorities
What to watch
bullSeverity of possible defectsbullProbability of possible defects
bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity
21
144 Fundamental test process ndash evaluating exit criteria and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 22110
144 Fundamental test process evaluating exit criteria andreporting
Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed
bull Check if testing mission should be changed
Test report ing
bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity
(see Rex Black (b4) amp RUP- Test discipline(s5))
22
1 4 5 F d l l
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 23110
145 Fundamental test process ndash test closure
bull Verify if test deliverables have been delivered
bull Check and close the remaining active bug reports
bull Archiving the test-ware and environment
bull Handover of the test environment
bull Analyze the identified test process problems (lessons learned)
bull Implement action plan based improvements
(see Rex Black (b4))
23
1 5 Th h l f i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 24110
15 The psychology of testing
24
Testing is regarded as a lsquodestructiversquo activity
(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts
bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise
bullShould be a planned organised kind ofperson
Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices
Rex BlackrsquosTop 10 professional errors
bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations
(see also Brian Marickrsquos article )
1 5 Th h l g f t ti g
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 25110
15 The psychology of testing
25
ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)
ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects
bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs
Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)
Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts
bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence
2 1 1 The V testing model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 26110
211 The V testing model
26
211 The V testing model ndash Verification amp Validationifi i C fi i b
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 27110
27
Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled
Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled
Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level
2 1 1 The W testing model ndash dynamic testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 28110
211 The W testing model ndash dynamic testing
28
2 1 1 The W testing model ndash static testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 29110
211 The W testing model static testing
29
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 30110
212 Software development models Waterfall
30
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 31110
212 Software development models Waterfall
Waterfall weaknesses
middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule
middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover
middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen
middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to
partition the system for delivery of pieces of the system
31
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 32110
p p yp
32
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 33110
p p yp
Rapid Prototype Model weaknesses
middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked
middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products
middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations
middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary
middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study
33
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 34110
34
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 35110
35
Incremental Model weaknesses
middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments
middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-
defined interfaces are required
middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 36110
36
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 37110
Spiral Model weaknesses
middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to
proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk
analysis and prototyping may be excessive
37
212 Software development models - Rational Unified Process
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 38110
38
213 Software development models ndash Testing life cycle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 39110
bull For each software activity there must be a corresponding testing activity
bull The objectives of the testing are specific to that ldquotestedrdquo activity
bull Plan analysis and design of a testing activity should be done during thecorresponding development activity
bull Review and inspection must be considered as part of testing activities
39
221 Test levels ndash Component testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 3110
111 Why is testing necessary - Testing goals and objectives
The main goals of testing
bull Find defectsbull Assess the level of quality of the software product and providing relatedinformation to the stakeholdersbull Prevent defectsbull Reduce risk of operational incidents
bull Increase the product qualityDifferent viewpoints and objectives
bull Unit amp integration test ndash find as many defects as possiblebull Acceptance testing ndash confirm that the system works as specified and that thequality is good enoughbull Testing metrics gathering ndash provide information to the project manager aboutthe product quality and the risks involved
bull Design tests early and review requirements ndash help prevent defects3
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 4110
112 Why is testing necessary - Testing Glossary
A programmer (or analyst) can make an error (mistake) which produces a defect(fault bug) in the programrsquos code If such a defect in the code is executed thesystem will fail to do what it should do (or it will do something it should not do)causing a failure
Error (mistake) = a human action that produces an incorrect resultDefect (bug) = a flaw that can cause the component or system to fail to perform its
required functionFailure = deviation of the component or system from its expected delivery serviceor result
Anomaly = any condition that deviates from expectations based on requirementsspecifications design documents user documentation standards or someonersquosperceptions or expectations
Defect masking = An occurrence in which one defect prevents the detection ofanother
4
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 5110
112 Why is testing necessary ndash Causes of the errorsDefects are caused by human errors
Why Because ofTime pressure - the more pressure we are under the more likely we are tomake mistakesbullCode complexity or new technology
bullToo many system interactionsbullRequirements not clearly defined changed amp not properly documentedbullWe make wrong assumptions about missing bits of informationbullPoor communicationbullPoor training
5
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 6110
112 Why is testing necessary - Causes of software defects -Defects taxonomy
6
(Boris Beizer)
bull Requirements (incorrect logic completeness verifiability documentation changes)bull Features and functionality (correctness missing case domain and boundarymessages exception mishandled)
bull Structural (control flow sequence data processing)bull Data (definition structure access handling)bull Implementation and Codingbull Integration (internal and external interfaces)bull System (architecture performance recovery partitioning environment)bull Test definition and execution (test design test execution documentation reporting)
(Cem Kaner (b1))bull User interface (functionality communication missing performance output)bull Error handling (prevention detection recovery)bull Boundary (numeric loops)
bull Calculation (wrong constants wrong operation order over amp underflow)bull Initialization (data item string loop control)bull Control flow (stop crash loop if-then-elsehellip)bull Data handling (data type parameter list values)bull Race amp Load conditions (event sequence no resources)bull Source and version control (old bug reappear)bull Testing (fail to notice fail to test fail to report)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 7110
113 Why is testing necessary - The role of testing in thesoftware life cycle
Testers do cooperate with Analysts ndash to review the specifications for completeness and correctnessensure that they are testablebull Designers ndash to improve interfaces testability and usabilitybull Programmers ndash to review the code and assess structural flawsbull Project manager ndash to estimate plan develop test cases perform tests andreport bugs to assess the quality and risksbull Quality assurance staff ndash to provide defect metricsbull Interactions with these project roles are very complex
RACI matrix (Responsible Accountable Consulted Informed)
7
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 8110
114 Why is testing necessary ndash What is quality
8
Quality (ISO) = The totality of the characteristics of an entity that bear on its
ability to satisfy stated or implied needs
bull there are many more definitions
Testing means not only to Verify (the thing is done right)hellip but also to Validate (the right thing is done)
Software quality includes reliability usability efficiency maintainability andportabilityRELIABILITY The ability of the software product to perform its requiredfunctions under stated conditions for a specified period of time or for aspecified number of operationsUSABILITY The capability of the software to be understood learned used
and attractive to the user when used under specified conditionsEFFICIENCY The capability of the software product to provide appropriateperformance relative to the amount of resources used under stated conditionsMAINTAINABILITY The ease with which a software product can be modifiedto correct defects modified to meet new requirements modified to make futuremaintenance easier or adapted to a changed environmentPORTABILITY The ease with which the software product can be transferredfrom one hardware or software environment to another
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 9110
114 Why is testing necessary - Testing and quality
Testing does not inject Quality
into the product but will measurethe level of the Quality of theproduct
Quality can be measured forbull progressbull variance (planned versus actual)
Measuring product quality bullFunctional compliance - functional software requirements testing
bullNon functional requirementsbullTest coverage criteriabullDefect count or defect trend criteria
9
1 1 4 Wh i i li ib
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 10110
114 Why is testing necessary ndash quality attributes
The QUINT model (extended ISO model)
10
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 11110
115 Why is testing necessary - How much testing is enough
when to stop testingThe five basic criteria often used to decide are
bull Previously defined coverage goals have been metbull The defect discovery rate has dropped below a previously defined thresholdbull The cost of finding the next defect exceeds the expected loss from that defectbull The project team reaches consensus that it is appropriate to release the productbull The manager decides to deliver the product
All these criteria are risk basedIt is important not depending on only one stopping criterion
Software Reliability Engineering can help also to determine when to stop testingby taking into consideration aspects like failure intensity
11
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 12110
12 What is testing - Defini tion of testing
12
Testing = the process concerned with planning the necessary static and dynamic
activities preparation and evaluation of software products and related deliverablesin order tobull determine that they satisfy specified requirementsbull demonstrate that they are fit for the intended usebull detect defects help and motivate the developers to fix thembull measure assess and improve the quality of the software product
Testing should be performed throughout the whole software life cycle
There are two basic types of testing execution and non-execution based
Other definitions
bull(IEEE) Testing = the process of analyzing a software item to detect the differencesbetween existing and required conditions and to evaluate its featuresbull(Myers (b3)) Testing = the process of executing a program with the intent of findingerrorsbull(Craig amp Jaskiel (b5)) Testing = a concurrent lifecycle process of engineering usingand maintaining test-ware in order to measure and improve the quality of thesoftware being tested
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 13110
12 What is testing ndash Testing ldquo schoolsrdquo
13
Analytic School - testing is rigorous academic and technicalbullTesting is a branch of CSMathematicsbullTesting techniques must have a logic-mathematical formbullKey Question Which techniques should we usebullRequire precise and detailed specifications
Factory School - testing is a way to measure progress with emphasis on cost and
repeatable standardsbullTesting must be managed amp cost effectivebullTesting validates the product amp measures development progressbullKey Questions How can we measure whether wersquore making progress When will we be donebullRequire clear boundaries between testing and other activities (startstop criteria)
bullEncourage standards (V-model) ldquobest practicesrdquo and certificationQuality School - emphasizes process amp quality acting as the gatekeeper bullSoftware quality requires disciplinebullTesters may need to police developers to follow the rulesbullKey Question Are we following a good process
bullTesting is a stepping stone to ldquoprocess improvementrdquoContext-Driven School - emphasizes people setting out to find the bugs that will bemost important to stakeholders
bullSoftware is created by people People set the contextbullTesting finds bugs acting as a skilled mental activitybullKey Question What testing would be most valuable right nowbullExpect changes Adapt testing plans based on test resultsbullTesting research requires empirical and psychological study
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 14110
13 General testing principles
14
bull Testing shows presence of defects but cannot prove that there are no moredefects testing can only reduce the probability of undiscovered defects
bull Complete exhaustive testing is impossible good strategy and risk managementmust be used
bull Pareto rule (defect clustering) usually 20 of the modules contain 80 of thebugs
bull Early testing testing activities should start as soon as possible (including hereplanning design reviews)
bull Pesticide paradox if the same set of tests are repeated over again no new bugswill be found the test cases should be reviewed modified and new test casesdeveloped
bull Context dependence test design and execution is context dependent (desktopweb applications real-time hellip)
bull Verification and Validation discovering defects cannot help a product that is not fitto the users needs
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 15110
13 General testing principles ndash heuristics of software testing
bullOperability - The better it works the more efficiently it can be tested
bullObservability - What we see is what we test
bull Controllability - The better we control the software the more the testing processcan be automated and optimized
bull Decomposability - By controlling the scope of testing we can quickly isolateproblems and perform effective and efficient testing
bull Simplicity - The less there is to test the more quickly we can test it
bull Stability - The fewer the changes the fewer are the disruptions to testing
bull Understandability - The more information we will have the smarter we will test
bull Suitability - The more we know about the intended use of the software the
better we can organize our testing to find important bugs15
1 4 1 F d t l t t h
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 16110
141 Fundamental test process ndash phases
-Test Planning amp Test control
-Test Analysis amp Design
-Test Implementation ampExecution
-Evaluating exit criteria ampReporting
-Test Closure activities
16
1 4 1 Fundamental test process planning amp control
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 17110
141 Fundamental test process ndash planning amp control
Planning1 Determine scope
bull Study project documents used software life-cycle specifications product desiredquality attributesbull Clarify test process expectations
2 Determine risksbull Choose quality risk analysis method (eg FMEA)
bull Document the list of risks probability impact priority identify mitigation actions3 Estimate testing effort determine costs develop schedule
bull Define necessary rolesbull Decompose test project into phases and tasks (WBS)bull Schedule tasks assign resources set-up dependencies
4 Refine planbull Select test strategy (how to do it what test types at which test levels)bull Select metrics to be used for defect tracking coverage monitoringbull Define entry and exit criteria
Controlbull Measure and analyze resultsbull Monitor testing progress coverage exit criteriabull Assign or reallocate resources update the test plan schedulebull Initiate corrective actionsbull Make decisions
17
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 18110
142 Fundamental test process ndash analysis amp design
bull Reviewing the test basis (such as requirements architecture designinterfaces)
bull Identifying test conditions or test requirements and required test data
based on analysis of test items and the specificationbull Designing the testsbull Choose test techniquesbull Identify test scenarios pre-conditions expected results post-
conditionsbull Identify possible test Oracles
bull Evaluating testability of the requirements and systembull Designing the test environment set-up and identifying any required
infrastructure and tools
(see Lee Copeland (b2))
18
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 19110
142 Fundamental test process ndash what is a test Oracle
19
bull The expected result (test outcome) must be defined at the test analysis stagebull Who will decide that (expected result = actual result) when the test will be
executed The Test Oracle
Test Oracle = A source to determine expected result a principle or mechanism torecognize a problem The Test Oracle can bebull an existing system (the old versionhellip)bull a document (specification user manual)bull a competent client representative
hellip but never the source code itself Oracles in use = Simplification of Risk do not assess lsquopass - failrsquo but instead
lsquoproblem - no problemrsquo
Problem Oracles and Automation - Our ability to automate testing isfundamentally constrained by our ability to create and use oracles
Possible issues
bull false alarmsbull missed bugs
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 20110
143 Fundamental test process ndash implementation amp execution
Test implementation
bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)
bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission
(see Rex Black (b4))
20
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 21110
143 Fundamental test process ndash priori tizing the Test Cases
Why prioritize the Test Cases
bullIt is not possible to test everything we must do our best in the time available
bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence
bullThis means we must prioritise and focus testing on the priorities
What to watch
bullSeverity of possible defectsbullProbability of possible defects
bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity
21
144 Fundamental test process ndash evaluating exit criteria and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 22110
144 Fundamental test process evaluating exit criteria andreporting
Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed
bull Check if testing mission should be changed
Test report ing
bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity
(see Rex Black (b4) amp RUP- Test discipline(s5))
22
1 4 5 F d l l
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 23110
145 Fundamental test process ndash test closure
bull Verify if test deliverables have been delivered
bull Check and close the remaining active bug reports
bull Archiving the test-ware and environment
bull Handover of the test environment
bull Analyze the identified test process problems (lessons learned)
bull Implement action plan based improvements
(see Rex Black (b4))
23
1 5 Th h l f i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 24110
15 The psychology of testing
24
Testing is regarded as a lsquodestructiversquo activity
(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts
bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise
bullShould be a planned organised kind ofperson
Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices
Rex BlackrsquosTop 10 professional errors
bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations
(see also Brian Marickrsquos article )
1 5 Th h l g f t ti g
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 25110
15 The psychology of testing
25
ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)
ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects
bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs
Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)
Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts
bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence
2 1 1 The V testing model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 26110
211 The V testing model
26
211 The V testing model ndash Verification amp Validationifi i C fi i b
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 27110
27
Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled
Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled
Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level
2 1 1 The W testing model ndash dynamic testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 28110
211 The W testing model ndash dynamic testing
28
2 1 1 The W testing model ndash static testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 29110
211 The W testing model static testing
29
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 30110
212 Software development models Waterfall
30
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 31110
212 Software development models Waterfall
Waterfall weaknesses
middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule
middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover
middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen
middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to
partition the system for delivery of pieces of the system
31
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 32110
p p yp
32
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 33110
p p yp
Rapid Prototype Model weaknesses
middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked
middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products
middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations
middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary
middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study
33
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 34110
34
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 35110
35
Incremental Model weaknesses
middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments
middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-
defined interfaces are required
middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 36110
36
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 37110
Spiral Model weaknesses
middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to
proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk
analysis and prototyping may be excessive
37
212 Software development models - Rational Unified Process
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 38110
38
213 Software development models ndash Testing life cycle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 39110
bull For each software activity there must be a corresponding testing activity
bull The objectives of the testing are specific to that ldquotestedrdquo activity
bull Plan analysis and design of a testing activity should be done during thecorresponding development activity
bull Review and inspection must be considered as part of testing activities
39
221 Test levels ndash Component testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 4110
112 Why is testing necessary - Testing Glossary
A programmer (or analyst) can make an error (mistake) which produces a defect(fault bug) in the programrsquos code If such a defect in the code is executed thesystem will fail to do what it should do (or it will do something it should not do)causing a failure
Error (mistake) = a human action that produces an incorrect resultDefect (bug) = a flaw that can cause the component or system to fail to perform its
required functionFailure = deviation of the component or system from its expected delivery serviceor result
Anomaly = any condition that deviates from expectations based on requirementsspecifications design documents user documentation standards or someonersquosperceptions or expectations
Defect masking = An occurrence in which one defect prevents the detection ofanother
4
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 5110
112 Why is testing necessary ndash Causes of the errorsDefects are caused by human errors
Why Because ofTime pressure - the more pressure we are under the more likely we are tomake mistakesbullCode complexity or new technology
bullToo many system interactionsbullRequirements not clearly defined changed amp not properly documentedbullWe make wrong assumptions about missing bits of informationbullPoor communicationbullPoor training
5
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 6110
112 Why is testing necessary - Causes of software defects -Defects taxonomy
6
(Boris Beizer)
bull Requirements (incorrect logic completeness verifiability documentation changes)bull Features and functionality (correctness missing case domain and boundarymessages exception mishandled)
bull Structural (control flow sequence data processing)bull Data (definition structure access handling)bull Implementation and Codingbull Integration (internal and external interfaces)bull System (architecture performance recovery partitioning environment)bull Test definition and execution (test design test execution documentation reporting)
(Cem Kaner (b1))bull User interface (functionality communication missing performance output)bull Error handling (prevention detection recovery)bull Boundary (numeric loops)
bull Calculation (wrong constants wrong operation order over amp underflow)bull Initialization (data item string loop control)bull Control flow (stop crash loop if-then-elsehellip)bull Data handling (data type parameter list values)bull Race amp Load conditions (event sequence no resources)bull Source and version control (old bug reappear)bull Testing (fail to notice fail to test fail to report)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 7110
113 Why is testing necessary - The role of testing in thesoftware life cycle
Testers do cooperate with Analysts ndash to review the specifications for completeness and correctnessensure that they are testablebull Designers ndash to improve interfaces testability and usabilitybull Programmers ndash to review the code and assess structural flawsbull Project manager ndash to estimate plan develop test cases perform tests andreport bugs to assess the quality and risksbull Quality assurance staff ndash to provide defect metricsbull Interactions with these project roles are very complex
RACI matrix (Responsible Accountable Consulted Informed)
7
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 8110
114 Why is testing necessary ndash What is quality
8
Quality (ISO) = The totality of the characteristics of an entity that bear on its
ability to satisfy stated or implied needs
bull there are many more definitions
Testing means not only to Verify (the thing is done right)hellip but also to Validate (the right thing is done)
Software quality includes reliability usability efficiency maintainability andportabilityRELIABILITY The ability of the software product to perform its requiredfunctions under stated conditions for a specified period of time or for aspecified number of operationsUSABILITY The capability of the software to be understood learned used
and attractive to the user when used under specified conditionsEFFICIENCY The capability of the software product to provide appropriateperformance relative to the amount of resources used under stated conditionsMAINTAINABILITY The ease with which a software product can be modifiedto correct defects modified to meet new requirements modified to make futuremaintenance easier or adapted to a changed environmentPORTABILITY The ease with which the software product can be transferredfrom one hardware or software environment to another
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 9110
114 Why is testing necessary - Testing and quality
Testing does not inject Quality
into the product but will measurethe level of the Quality of theproduct
Quality can be measured forbull progressbull variance (planned versus actual)
Measuring product quality bullFunctional compliance - functional software requirements testing
bullNon functional requirementsbullTest coverage criteriabullDefect count or defect trend criteria
9
1 1 4 Wh i i li ib
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 10110
114 Why is testing necessary ndash quality attributes
The QUINT model (extended ISO model)
10
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 11110
115 Why is testing necessary - How much testing is enough
when to stop testingThe five basic criteria often used to decide are
bull Previously defined coverage goals have been metbull The defect discovery rate has dropped below a previously defined thresholdbull The cost of finding the next defect exceeds the expected loss from that defectbull The project team reaches consensus that it is appropriate to release the productbull The manager decides to deliver the product
All these criteria are risk basedIt is important not depending on only one stopping criterion
Software Reliability Engineering can help also to determine when to stop testingby taking into consideration aspects like failure intensity
11
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 12110
12 What is testing - Defini tion of testing
12
Testing = the process concerned with planning the necessary static and dynamic
activities preparation and evaluation of software products and related deliverablesin order tobull determine that they satisfy specified requirementsbull demonstrate that they are fit for the intended usebull detect defects help and motivate the developers to fix thembull measure assess and improve the quality of the software product
Testing should be performed throughout the whole software life cycle
There are two basic types of testing execution and non-execution based
Other definitions
bull(IEEE) Testing = the process of analyzing a software item to detect the differencesbetween existing and required conditions and to evaluate its featuresbull(Myers (b3)) Testing = the process of executing a program with the intent of findingerrorsbull(Craig amp Jaskiel (b5)) Testing = a concurrent lifecycle process of engineering usingand maintaining test-ware in order to measure and improve the quality of thesoftware being tested
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 13110
12 What is testing ndash Testing ldquo schoolsrdquo
13
Analytic School - testing is rigorous academic and technicalbullTesting is a branch of CSMathematicsbullTesting techniques must have a logic-mathematical formbullKey Question Which techniques should we usebullRequire precise and detailed specifications
Factory School - testing is a way to measure progress with emphasis on cost and
repeatable standardsbullTesting must be managed amp cost effectivebullTesting validates the product amp measures development progressbullKey Questions How can we measure whether wersquore making progress When will we be donebullRequire clear boundaries between testing and other activities (startstop criteria)
bullEncourage standards (V-model) ldquobest practicesrdquo and certificationQuality School - emphasizes process amp quality acting as the gatekeeper bullSoftware quality requires disciplinebullTesters may need to police developers to follow the rulesbullKey Question Are we following a good process
bullTesting is a stepping stone to ldquoprocess improvementrdquoContext-Driven School - emphasizes people setting out to find the bugs that will bemost important to stakeholders
bullSoftware is created by people People set the contextbullTesting finds bugs acting as a skilled mental activitybullKey Question What testing would be most valuable right nowbullExpect changes Adapt testing plans based on test resultsbullTesting research requires empirical and psychological study
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 14110
13 General testing principles
14
bull Testing shows presence of defects but cannot prove that there are no moredefects testing can only reduce the probability of undiscovered defects
bull Complete exhaustive testing is impossible good strategy and risk managementmust be used
bull Pareto rule (defect clustering) usually 20 of the modules contain 80 of thebugs
bull Early testing testing activities should start as soon as possible (including hereplanning design reviews)
bull Pesticide paradox if the same set of tests are repeated over again no new bugswill be found the test cases should be reviewed modified and new test casesdeveloped
bull Context dependence test design and execution is context dependent (desktopweb applications real-time hellip)
bull Verification and Validation discovering defects cannot help a product that is not fitto the users needs
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 15110
13 General testing principles ndash heuristics of software testing
bullOperability - The better it works the more efficiently it can be tested
bullObservability - What we see is what we test
bull Controllability - The better we control the software the more the testing processcan be automated and optimized
bull Decomposability - By controlling the scope of testing we can quickly isolateproblems and perform effective and efficient testing
bull Simplicity - The less there is to test the more quickly we can test it
bull Stability - The fewer the changes the fewer are the disruptions to testing
bull Understandability - The more information we will have the smarter we will test
bull Suitability - The more we know about the intended use of the software the
better we can organize our testing to find important bugs15
1 4 1 F d t l t t h
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 16110
141 Fundamental test process ndash phases
-Test Planning amp Test control
-Test Analysis amp Design
-Test Implementation ampExecution
-Evaluating exit criteria ampReporting
-Test Closure activities
16
1 4 1 Fundamental test process planning amp control
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 17110
141 Fundamental test process ndash planning amp control
Planning1 Determine scope
bull Study project documents used software life-cycle specifications product desiredquality attributesbull Clarify test process expectations
2 Determine risksbull Choose quality risk analysis method (eg FMEA)
bull Document the list of risks probability impact priority identify mitigation actions3 Estimate testing effort determine costs develop schedule
bull Define necessary rolesbull Decompose test project into phases and tasks (WBS)bull Schedule tasks assign resources set-up dependencies
4 Refine planbull Select test strategy (how to do it what test types at which test levels)bull Select metrics to be used for defect tracking coverage monitoringbull Define entry and exit criteria
Controlbull Measure and analyze resultsbull Monitor testing progress coverage exit criteriabull Assign or reallocate resources update the test plan schedulebull Initiate corrective actionsbull Make decisions
17
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 18110
142 Fundamental test process ndash analysis amp design
bull Reviewing the test basis (such as requirements architecture designinterfaces)
bull Identifying test conditions or test requirements and required test data
based on analysis of test items and the specificationbull Designing the testsbull Choose test techniquesbull Identify test scenarios pre-conditions expected results post-
conditionsbull Identify possible test Oracles
bull Evaluating testability of the requirements and systembull Designing the test environment set-up and identifying any required
infrastructure and tools
(see Lee Copeland (b2))
18
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 19110
142 Fundamental test process ndash what is a test Oracle
19
bull The expected result (test outcome) must be defined at the test analysis stagebull Who will decide that (expected result = actual result) when the test will be
executed The Test Oracle
Test Oracle = A source to determine expected result a principle or mechanism torecognize a problem The Test Oracle can bebull an existing system (the old versionhellip)bull a document (specification user manual)bull a competent client representative
hellip but never the source code itself Oracles in use = Simplification of Risk do not assess lsquopass - failrsquo but instead
lsquoproblem - no problemrsquo
Problem Oracles and Automation - Our ability to automate testing isfundamentally constrained by our ability to create and use oracles
Possible issues
bull false alarmsbull missed bugs
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 20110
143 Fundamental test process ndash implementation amp execution
Test implementation
bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)
bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission
(see Rex Black (b4))
20
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 21110
143 Fundamental test process ndash priori tizing the Test Cases
Why prioritize the Test Cases
bullIt is not possible to test everything we must do our best in the time available
bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence
bullThis means we must prioritise and focus testing on the priorities
What to watch
bullSeverity of possible defectsbullProbability of possible defects
bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity
21
144 Fundamental test process ndash evaluating exit criteria and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 22110
144 Fundamental test process evaluating exit criteria andreporting
Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed
bull Check if testing mission should be changed
Test report ing
bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity
(see Rex Black (b4) amp RUP- Test discipline(s5))
22
1 4 5 F d l l
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 23110
145 Fundamental test process ndash test closure
bull Verify if test deliverables have been delivered
bull Check and close the remaining active bug reports
bull Archiving the test-ware and environment
bull Handover of the test environment
bull Analyze the identified test process problems (lessons learned)
bull Implement action plan based improvements
(see Rex Black (b4))
23
1 5 Th h l f i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 24110
15 The psychology of testing
24
Testing is regarded as a lsquodestructiversquo activity
(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts
bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise
bullShould be a planned organised kind ofperson
Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices
Rex BlackrsquosTop 10 professional errors
bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations
(see also Brian Marickrsquos article )
1 5 Th h l g f t ti g
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 25110
15 The psychology of testing
25
ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)
ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects
bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs
Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)
Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts
bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence
2 1 1 The V testing model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 26110
211 The V testing model
26
211 The V testing model ndash Verification amp Validationifi i C fi i b
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 27110
27
Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled
Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled
Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level
2 1 1 The W testing model ndash dynamic testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 28110
211 The W testing model ndash dynamic testing
28
2 1 1 The W testing model ndash static testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 29110
211 The W testing model static testing
29
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 30110
212 Software development models Waterfall
30
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 31110
212 Software development models Waterfall
Waterfall weaknesses
middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule
middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover
middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen
middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to
partition the system for delivery of pieces of the system
31
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 32110
p p yp
32
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 33110
p p yp
Rapid Prototype Model weaknesses
middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked
middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products
middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations
middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary
middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study
33
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 34110
34
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 35110
35
Incremental Model weaknesses
middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments
middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-
defined interfaces are required
middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 36110
36
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 37110
Spiral Model weaknesses
middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to
proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk
analysis and prototyping may be excessive
37
212 Software development models - Rational Unified Process
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 38110
38
213 Software development models ndash Testing life cycle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 39110
bull For each software activity there must be a corresponding testing activity
bull The objectives of the testing are specific to that ldquotestedrdquo activity
bull Plan analysis and design of a testing activity should be done during thecorresponding development activity
bull Review and inspection must be considered as part of testing activities
39
221 Test levels ndash Component testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 5110
112 Why is testing necessary ndash Causes of the errorsDefects are caused by human errors
Why Because ofTime pressure - the more pressure we are under the more likely we are tomake mistakesbullCode complexity or new technology
bullToo many system interactionsbullRequirements not clearly defined changed amp not properly documentedbullWe make wrong assumptions about missing bits of informationbullPoor communicationbullPoor training
5
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 6110
112 Why is testing necessary - Causes of software defects -Defects taxonomy
6
(Boris Beizer)
bull Requirements (incorrect logic completeness verifiability documentation changes)bull Features and functionality (correctness missing case domain and boundarymessages exception mishandled)
bull Structural (control flow sequence data processing)bull Data (definition structure access handling)bull Implementation and Codingbull Integration (internal and external interfaces)bull System (architecture performance recovery partitioning environment)bull Test definition and execution (test design test execution documentation reporting)
(Cem Kaner (b1))bull User interface (functionality communication missing performance output)bull Error handling (prevention detection recovery)bull Boundary (numeric loops)
bull Calculation (wrong constants wrong operation order over amp underflow)bull Initialization (data item string loop control)bull Control flow (stop crash loop if-then-elsehellip)bull Data handling (data type parameter list values)bull Race amp Load conditions (event sequence no resources)bull Source and version control (old bug reappear)bull Testing (fail to notice fail to test fail to report)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 7110
113 Why is testing necessary - The role of testing in thesoftware life cycle
Testers do cooperate with Analysts ndash to review the specifications for completeness and correctnessensure that they are testablebull Designers ndash to improve interfaces testability and usabilitybull Programmers ndash to review the code and assess structural flawsbull Project manager ndash to estimate plan develop test cases perform tests andreport bugs to assess the quality and risksbull Quality assurance staff ndash to provide defect metricsbull Interactions with these project roles are very complex
RACI matrix (Responsible Accountable Consulted Informed)
7
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 8110
114 Why is testing necessary ndash What is quality
8
Quality (ISO) = The totality of the characteristics of an entity that bear on its
ability to satisfy stated or implied needs
bull there are many more definitions
Testing means not only to Verify (the thing is done right)hellip but also to Validate (the right thing is done)
Software quality includes reliability usability efficiency maintainability andportabilityRELIABILITY The ability of the software product to perform its requiredfunctions under stated conditions for a specified period of time or for aspecified number of operationsUSABILITY The capability of the software to be understood learned used
and attractive to the user when used under specified conditionsEFFICIENCY The capability of the software product to provide appropriateperformance relative to the amount of resources used under stated conditionsMAINTAINABILITY The ease with which a software product can be modifiedto correct defects modified to meet new requirements modified to make futuremaintenance easier or adapted to a changed environmentPORTABILITY The ease with which the software product can be transferredfrom one hardware or software environment to another
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 9110
114 Why is testing necessary - Testing and quality
Testing does not inject Quality
into the product but will measurethe level of the Quality of theproduct
Quality can be measured forbull progressbull variance (planned versus actual)
Measuring product quality bullFunctional compliance - functional software requirements testing
bullNon functional requirementsbullTest coverage criteriabullDefect count or defect trend criteria
9
1 1 4 Wh i i li ib
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 10110
114 Why is testing necessary ndash quality attributes
The QUINT model (extended ISO model)
10
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 11110
115 Why is testing necessary - How much testing is enough
when to stop testingThe five basic criteria often used to decide are
bull Previously defined coverage goals have been metbull The defect discovery rate has dropped below a previously defined thresholdbull The cost of finding the next defect exceeds the expected loss from that defectbull The project team reaches consensus that it is appropriate to release the productbull The manager decides to deliver the product
All these criteria are risk basedIt is important not depending on only one stopping criterion
Software Reliability Engineering can help also to determine when to stop testingby taking into consideration aspects like failure intensity
11
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 12110
12 What is testing - Defini tion of testing
12
Testing = the process concerned with planning the necessary static and dynamic
activities preparation and evaluation of software products and related deliverablesin order tobull determine that they satisfy specified requirementsbull demonstrate that they are fit for the intended usebull detect defects help and motivate the developers to fix thembull measure assess and improve the quality of the software product
Testing should be performed throughout the whole software life cycle
There are two basic types of testing execution and non-execution based
Other definitions
bull(IEEE) Testing = the process of analyzing a software item to detect the differencesbetween existing and required conditions and to evaluate its featuresbull(Myers (b3)) Testing = the process of executing a program with the intent of findingerrorsbull(Craig amp Jaskiel (b5)) Testing = a concurrent lifecycle process of engineering usingand maintaining test-ware in order to measure and improve the quality of thesoftware being tested
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 13110
12 What is testing ndash Testing ldquo schoolsrdquo
13
Analytic School - testing is rigorous academic and technicalbullTesting is a branch of CSMathematicsbullTesting techniques must have a logic-mathematical formbullKey Question Which techniques should we usebullRequire precise and detailed specifications
Factory School - testing is a way to measure progress with emphasis on cost and
repeatable standardsbullTesting must be managed amp cost effectivebullTesting validates the product amp measures development progressbullKey Questions How can we measure whether wersquore making progress When will we be donebullRequire clear boundaries between testing and other activities (startstop criteria)
bullEncourage standards (V-model) ldquobest practicesrdquo and certificationQuality School - emphasizes process amp quality acting as the gatekeeper bullSoftware quality requires disciplinebullTesters may need to police developers to follow the rulesbullKey Question Are we following a good process
bullTesting is a stepping stone to ldquoprocess improvementrdquoContext-Driven School - emphasizes people setting out to find the bugs that will bemost important to stakeholders
bullSoftware is created by people People set the contextbullTesting finds bugs acting as a skilled mental activitybullKey Question What testing would be most valuable right nowbullExpect changes Adapt testing plans based on test resultsbullTesting research requires empirical and psychological study
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 14110
13 General testing principles
14
bull Testing shows presence of defects but cannot prove that there are no moredefects testing can only reduce the probability of undiscovered defects
bull Complete exhaustive testing is impossible good strategy and risk managementmust be used
bull Pareto rule (defect clustering) usually 20 of the modules contain 80 of thebugs
bull Early testing testing activities should start as soon as possible (including hereplanning design reviews)
bull Pesticide paradox if the same set of tests are repeated over again no new bugswill be found the test cases should be reviewed modified and new test casesdeveloped
bull Context dependence test design and execution is context dependent (desktopweb applications real-time hellip)
bull Verification and Validation discovering defects cannot help a product that is not fitto the users needs
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 15110
13 General testing principles ndash heuristics of software testing
bullOperability - The better it works the more efficiently it can be tested
bullObservability - What we see is what we test
bull Controllability - The better we control the software the more the testing processcan be automated and optimized
bull Decomposability - By controlling the scope of testing we can quickly isolateproblems and perform effective and efficient testing
bull Simplicity - The less there is to test the more quickly we can test it
bull Stability - The fewer the changes the fewer are the disruptions to testing
bull Understandability - The more information we will have the smarter we will test
bull Suitability - The more we know about the intended use of the software the
better we can organize our testing to find important bugs15
1 4 1 F d t l t t h
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 16110
141 Fundamental test process ndash phases
-Test Planning amp Test control
-Test Analysis amp Design
-Test Implementation ampExecution
-Evaluating exit criteria ampReporting
-Test Closure activities
16
1 4 1 Fundamental test process planning amp control
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 17110
141 Fundamental test process ndash planning amp control
Planning1 Determine scope
bull Study project documents used software life-cycle specifications product desiredquality attributesbull Clarify test process expectations
2 Determine risksbull Choose quality risk analysis method (eg FMEA)
bull Document the list of risks probability impact priority identify mitigation actions3 Estimate testing effort determine costs develop schedule
bull Define necessary rolesbull Decompose test project into phases and tasks (WBS)bull Schedule tasks assign resources set-up dependencies
4 Refine planbull Select test strategy (how to do it what test types at which test levels)bull Select metrics to be used for defect tracking coverage monitoringbull Define entry and exit criteria
Controlbull Measure and analyze resultsbull Monitor testing progress coverage exit criteriabull Assign or reallocate resources update the test plan schedulebull Initiate corrective actionsbull Make decisions
17
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 18110
142 Fundamental test process ndash analysis amp design
bull Reviewing the test basis (such as requirements architecture designinterfaces)
bull Identifying test conditions or test requirements and required test data
based on analysis of test items and the specificationbull Designing the testsbull Choose test techniquesbull Identify test scenarios pre-conditions expected results post-
conditionsbull Identify possible test Oracles
bull Evaluating testability of the requirements and systembull Designing the test environment set-up and identifying any required
infrastructure and tools
(see Lee Copeland (b2))
18
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 19110
142 Fundamental test process ndash what is a test Oracle
19
bull The expected result (test outcome) must be defined at the test analysis stagebull Who will decide that (expected result = actual result) when the test will be
executed The Test Oracle
Test Oracle = A source to determine expected result a principle or mechanism torecognize a problem The Test Oracle can bebull an existing system (the old versionhellip)bull a document (specification user manual)bull a competent client representative
hellip but never the source code itself Oracles in use = Simplification of Risk do not assess lsquopass - failrsquo but instead
lsquoproblem - no problemrsquo
Problem Oracles and Automation - Our ability to automate testing isfundamentally constrained by our ability to create and use oracles
Possible issues
bull false alarmsbull missed bugs
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 20110
143 Fundamental test process ndash implementation amp execution
Test implementation
bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)
bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission
(see Rex Black (b4))
20
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 21110
143 Fundamental test process ndash priori tizing the Test Cases
Why prioritize the Test Cases
bullIt is not possible to test everything we must do our best in the time available
bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence
bullThis means we must prioritise and focus testing on the priorities
What to watch
bullSeverity of possible defectsbullProbability of possible defects
bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity
21
144 Fundamental test process ndash evaluating exit criteria and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 22110
144 Fundamental test process evaluating exit criteria andreporting
Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed
bull Check if testing mission should be changed
Test report ing
bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity
(see Rex Black (b4) amp RUP- Test discipline(s5))
22
1 4 5 F d l l
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 23110
145 Fundamental test process ndash test closure
bull Verify if test deliverables have been delivered
bull Check and close the remaining active bug reports
bull Archiving the test-ware and environment
bull Handover of the test environment
bull Analyze the identified test process problems (lessons learned)
bull Implement action plan based improvements
(see Rex Black (b4))
23
1 5 Th h l f i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 24110
15 The psychology of testing
24
Testing is regarded as a lsquodestructiversquo activity
(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts
bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise
bullShould be a planned organised kind ofperson
Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices
Rex BlackrsquosTop 10 professional errors
bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations
(see also Brian Marickrsquos article )
1 5 Th h l g f t ti g
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 25110
15 The psychology of testing
25
ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)
ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects
bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs
Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)
Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts
bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence
2 1 1 The V testing model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 26110
211 The V testing model
26
211 The V testing model ndash Verification amp Validationifi i C fi i b
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 27110
27
Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled
Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled
Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level
2 1 1 The W testing model ndash dynamic testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 28110
211 The W testing model ndash dynamic testing
28
2 1 1 The W testing model ndash static testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 29110
211 The W testing model static testing
29
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 30110
212 Software development models Waterfall
30
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 31110
212 Software development models Waterfall
Waterfall weaknesses
middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule
middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover
middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen
middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to
partition the system for delivery of pieces of the system
31
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 32110
p p yp
32
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 33110
p p yp
Rapid Prototype Model weaknesses
middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked
middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products
middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations
middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary
middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study
33
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 34110
34
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 35110
35
Incremental Model weaknesses
middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments
middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-
defined interfaces are required
middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 36110
36
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 37110
Spiral Model weaknesses
middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to
proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk
analysis and prototyping may be excessive
37
212 Software development models - Rational Unified Process
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 38110
38
213 Software development models ndash Testing life cycle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 39110
bull For each software activity there must be a corresponding testing activity
bull The objectives of the testing are specific to that ldquotestedrdquo activity
bull Plan analysis and design of a testing activity should be done during thecorresponding development activity
bull Review and inspection must be considered as part of testing activities
39
221 Test levels ndash Component testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 6110
112 Why is testing necessary - Causes of software defects -Defects taxonomy
6
(Boris Beizer)
bull Requirements (incorrect logic completeness verifiability documentation changes)bull Features and functionality (correctness missing case domain and boundarymessages exception mishandled)
bull Structural (control flow sequence data processing)bull Data (definition structure access handling)bull Implementation and Codingbull Integration (internal and external interfaces)bull System (architecture performance recovery partitioning environment)bull Test definition and execution (test design test execution documentation reporting)
(Cem Kaner (b1))bull User interface (functionality communication missing performance output)bull Error handling (prevention detection recovery)bull Boundary (numeric loops)
bull Calculation (wrong constants wrong operation order over amp underflow)bull Initialization (data item string loop control)bull Control flow (stop crash loop if-then-elsehellip)bull Data handling (data type parameter list values)bull Race amp Load conditions (event sequence no resources)bull Source and version control (old bug reappear)bull Testing (fail to notice fail to test fail to report)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 7110
113 Why is testing necessary - The role of testing in thesoftware life cycle
Testers do cooperate with Analysts ndash to review the specifications for completeness and correctnessensure that they are testablebull Designers ndash to improve interfaces testability and usabilitybull Programmers ndash to review the code and assess structural flawsbull Project manager ndash to estimate plan develop test cases perform tests andreport bugs to assess the quality and risksbull Quality assurance staff ndash to provide defect metricsbull Interactions with these project roles are very complex
RACI matrix (Responsible Accountable Consulted Informed)
7
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 8110
114 Why is testing necessary ndash What is quality
8
Quality (ISO) = The totality of the characteristics of an entity that bear on its
ability to satisfy stated or implied needs
bull there are many more definitions
Testing means not only to Verify (the thing is done right)hellip but also to Validate (the right thing is done)
Software quality includes reliability usability efficiency maintainability andportabilityRELIABILITY The ability of the software product to perform its requiredfunctions under stated conditions for a specified period of time or for aspecified number of operationsUSABILITY The capability of the software to be understood learned used
and attractive to the user when used under specified conditionsEFFICIENCY The capability of the software product to provide appropriateperformance relative to the amount of resources used under stated conditionsMAINTAINABILITY The ease with which a software product can be modifiedto correct defects modified to meet new requirements modified to make futuremaintenance easier or adapted to a changed environmentPORTABILITY The ease with which the software product can be transferredfrom one hardware or software environment to another
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 9110
114 Why is testing necessary - Testing and quality
Testing does not inject Quality
into the product but will measurethe level of the Quality of theproduct
Quality can be measured forbull progressbull variance (planned versus actual)
Measuring product quality bullFunctional compliance - functional software requirements testing
bullNon functional requirementsbullTest coverage criteriabullDefect count or defect trend criteria
9
1 1 4 Wh i i li ib
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 10110
114 Why is testing necessary ndash quality attributes
The QUINT model (extended ISO model)
10
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 11110
115 Why is testing necessary - How much testing is enough
when to stop testingThe five basic criteria often used to decide are
bull Previously defined coverage goals have been metbull The defect discovery rate has dropped below a previously defined thresholdbull The cost of finding the next defect exceeds the expected loss from that defectbull The project team reaches consensus that it is appropriate to release the productbull The manager decides to deliver the product
All these criteria are risk basedIt is important not depending on only one stopping criterion
Software Reliability Engineering can help also to determine when to stop testingby taking into consideration aspects like failure intensity
11
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 12110
12 What is testing - Defini tion of testing
12
Testing = the process concerned with planning the necessary static and dynamic
activities preparation and evaluation of software products and related deliverablesin order tobull determine that they satisfy specified requirementsbull demonstrate that they are fit for the intended usebull detect defects help and motivate the developers to fix thembull measure assess and improve the quality of the software product
Testing should be performed throughout the whole software life cycle
There are two basic types of testing execution and non-execution based
Other definitions
bull(IEEE) Testing = the process of analyzing a software item to detect the differencesbetween existing and required conditions and to evaluate its featuresbull(Myers (b3)) Testing = the process of executing a program with the intent of findingerrorsbull(Craig amp Jaskiel (b5)) Testing = a concurrent lifecycle process of engineering usingand maintaining test-ware in order to measure and improve the quality of thesoftware being tested
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 13110
12 What is testing ndash Testing ldquo schoolsrdquo
13
Analytic School - testing is rigorous academic and technicalbullTesting is a branch of CSMathematicsbullTesting techniques must have a logic-mathematical formbullKey Question Which techniques should we usebullRequire precise and detailed specifications
Factory School - testing is a way to measure progress with emphasis on cost and
repeatable standardsbullTesting must be managed amp cost effectivebullTesting validates the product amp measures development progressbullKey Questions How can we measure whether wersquore making progress When will we be donebullRequire clear boundaries between testing and other activities (startstop criteria)
bullEncourage standards (V-model) ldquobest practicesrdquo and certificationQuality School - emphasizes process amp quality acting as the gatekeeper bullSoftware quality requires disciplinebullTesters may need to police developers to follow the rulesbullKey Question Are we following a good process
bullTesting is a stepping stone to ldquoprocess improvementrdquoContext-Driven School - emphasizes people setting out to find the bugs that will bemost important to stakeholders
bullSoftware is created by people People set the contextbullTesting finds bugs acting as a skilled mental activitybullKey Question What testing would be most valuable right nowbullExpect changes Adapt testing plans based on test resultsbullTesting research requires empirical and psychological study
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 14110
13 General testing principles
14
bull Testing shows presence of defects but cannot prove that there are no moredefects testing can only reduce the probability of undiscovered defects
bull Complete exhaustive testing is impossible good strategy and risk managementmust be used
bull Pareto rule (defect clustering) usually 20 of the modules contain 80 of thebugs
bull Early testing testing activities should start as soon as possible (including hereplanning design reviews)
bull Pesticide paradox if the same set of tests are repeated over again no new bugswill be found the test cases should be reviewed modified and new test casesdeveloped
bull Context dependence test design and execution is context dependent (desktopweb applications real-time hellip)
bull Verification and Validation discovering defects cannot help a product that is not fitto the users needs
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 15110
13 General testing principles ndash heuristics of software testing
bullOperability - The better it works the more efficiently it can be tested
bullObservability - What we see is what we test
bull Controllability - The better we control the software the more the testing processcan be automated and optimized
bull Decomposability - By controlling the scope of testing we can quickly isolateproblems and perform effective and efficient testing
bull Simplicity - The less there is to test the more quickly we can test it
bull Stability - The fewer the changes the fewer are the disruptions to testing
bull Understandability - The more information we will have the smarter we will test
bull Suitability - The more we know about the intended use of the software the
better we can organize our testing to find important bugs15
1 4 1 F d t l t t h
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 16110
141 Fundamental test process ndash phases
-Test Planning amp Test control
-Test Analysis amp Design
-Test Implementation ampExecution
-Evaluating exit criteria ampReporting
-Test Closure activities
16
1 4 1 Fundamental test process planning amp control
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 17110
141 Fundamental test process ndash planning amp control
Planning1 Determine scope
bull Study project documents used software life-cycle specifications product desiredquality attributesbull Clarify test process expectations
2 Determine risksbull Choose quality risk analysis method (eg FMEA)
bull Document the list of risks probability impact priority identify mitigation actions3 Estimate testing effort determine costs develop schedule
bull Define necessary rolesbull Decompose test project into phases and tasks (WBS)bull Schedule tasks assign resources set-up dependencies
4 Refine planbull Select test strategy (how to do it what test types at which test levels)bull Select metrics to be used for defect tracking coverage monitoringbull Define entry and exit criteria
Controlbull Measure and analyze resultsbull Monitor testing progress coverage exit criteriabull Assign or reallocate resources update the test plan schedulebull Initiate corrective actionsbull Make decisions
17
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 18110
142 Fundamental test process ndash analysis amp design
bull Reviewing the test basis (such as requirements architecture designinterfaces)
bull Identifying test conditions or test requirements and required test data
based on analysis of test items and the specificationbull Designing the testsbull Choose test techniquesbull Identify test scenarios pre-conditions expected results post-
conditionsbull Identify possible test Oracles
bull Evaluating testability of the requirements and systembull Designing the test environment set-up and identifying any required
infrastructure and tools
(see Lee Copeland (b2))
18
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 19110
142 Fundamental test process ndash what is a test Oracle
19
bull The expected result (test outcome) must be defined at the test analysis stagebull Who will decide that (expected result = actual result) when the test will be
executed The Test Oracle
Test Oracle = A source to determine expected result a principle or mechanism torecognize a problem The Test Oracle can bebull an existing system (the old versionhellip)bull a document (specification user manual)bull a competent client representative
hellip but never the source code itself Oracles in use = Simplification of Risk do not assess lsquopass - failrsquo but instead
lsquoproblem - no problemrsquo
Problem Oracles and Automation - Our ability to automate testing isfundamentally constrained by our ability to create and use oracles
Possible issues
bull false alarmsbull missed bugs
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 20110
143 Fundamental test process ndash implementation amp execution
Test implementation
bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)
bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission
(see Rex Black (b4))
20
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 21110
143 Fundamental test process ndash priori tizing the Test Cases
Why prioritize the Test Cases
bullIt is not possible to test everything we must do our best in the time available
bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence
bullThis means we must prioritise and focus testing on the priorities
What to watch
bullSeverity of possible defectsbullProbability of possible defects
bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity
21
144 Fundamental test process ndash evaluating exit criteria and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 22110
144 Fundamental test process evaluating exit criteria andreporting
Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed
bull Check if testing mission should be changed
Test report ing
bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity
(see Rex Black (b4) amp RUP- Test discipline(s5))
22
1 4 5 F d l l
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 23110
145 Fundamental test process ndash test closure
bull Verify if test deliverables have been delivered
bull Check and close the remaining active bug reports
bull Archiving the test-ware and environment
bull Handover of the test environment
bull Analyze the identified test process problems (lessons learned)
bull Implement action plan based improvements
(see Rex Black (b4))
23
1 5 Th h l f i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 24110
15 The psychology of testing
24
Testing is regarded as a lsquodestructiversquo activity
(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts
bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise
bullShould be a planned organised kind ofperson
Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices
Rex BlackrsquosTop 10 professional errors
bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations
(see also Brian Marickrsquos article )
1 5 Th h l g f t ti g
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 25110
15 The psychology of testing
25
ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)
ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects
bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs
Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)
Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts
bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence
2 1 1 The V testing model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 26110
211 The V testing model
26
211 The V testing model ndash Verification amp Validationifi i C fi i b
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 27110
27
Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled
Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled
Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level
2 1 1 The W testing model ndash dynamic testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 28110
211 The W testing model ndash dynamic testing
28
2 1 1 The W testing model ndash static testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 29110
211 The W testing model static testing
29
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 30110
212 Software development models Waterfall
30
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 31110
212 Software development models Waterfall
Waterfall weaknesses
middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule
middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover
middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen
middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to
partition the system for delivery of pieces of the system
31
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 32110
p p yp
32
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 33110
p p yp
Rapid Prototype Model weaknesses
middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked
middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products
middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations
middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary
middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study
33
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 34110
34
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 35110
35
Incremental Model weaknesses
middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments
middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-
defined interfaces are required
middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 36110
36
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 37110
Spiral Model weaknesses
middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to
proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk
analysis and prototyping may be excessive
37
212 Software development models - Rational Unified Process
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 38110
38
213 Software development models ndash Testing life cycle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 39110
bull For each software activity there must be a corresponding testing activity
bull The objectives of the testing are specific to that ldquotestedrdquo activity
bull Plan analysis and design of a testing activity should be done during thecorresponding development activity
bull Review and inspection must be considered as part of testing activities
39
221 Test levels ndash Component testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 7110
113 Why is testing necessary - The role of testing in thesoftware life cycle
Testers do cooperate with Analysts ndash to review the specifications for completeness and correctnessensure that they are testablebull Designers ndash to improve interfaces testability and usabilitybull Programmers ndash to review the code and assess structural flawsbull Project manager ndash to estimate plan develop test cases perform tests andreport bugs to assess the quality and risksbull Quality assurance staff ndash to provide defect metricsbull Interactions with these project roles are very complex
RACI matrix (Responsible Accountable Consulted Informed)
7
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 8110
114 Why is testing necessary ndash What is quality
8
Quality (ISO) = The totality of the characteristics of an entity that bear on its
ability to satisfy stated or implied needs
bull there are many more definitions
Testing means not only to Verify (the thing is done right)hellip but also to Validate (the right thing is done)
Software quality includes reliability usability efficiency maintainability andportabilityRELIABILITY The ability of the software product to perform its requiredfunctions under stated conditions for a specified period of time or for aspecified number of operationsUSABILITY The capability of the software to be understood learned used
and attractive to the user when used under specified conditionsEFFICIENCY The capability of the software product to provide appropriateperformance relative to the amount of resources used under stated conditionsMAINTAINABILITY The ease with which a software product can be modifiedto correct defects modified to meet new requirements modified to make futuremaintenance easier or adapted to a changed environmentPORTABILITY The ease with which the software product can be transferredfrom one hardware or software environment to another
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 9110
114 Why is testing necessary - Testing and quality
Testing does not inject Quality
into the product but will measurethe level of the Quality of theproduct
Quality can be measured forbull progressbull variance (planned versus actual)
Measuring product quality bullFunctional compliance - functional software requirements testing
bullNon functional requirementsbullTest coverage criteriabullDefect count or defect trend criteria
9
1 1 4 Wh i i li ib
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 10110
114 Why is testing necessary ndash quality attributes
The QUINT model (extended ISO model)
10
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 11110
115 Why is testing necessary - How much testing is enough
when to stop testingThe five basic criteria often used to decide are
bull Previously defined coverage goals have been metbull The defect discovery rate has dropped below a previously defined thresholdbull The cost of finding the next defect exceeds the expected loss from that defectbull The project team reaches consensus that it is appropriate to release the productbull The manager decides to deliver the product
All these criteria are risk basedIt is important not depending on only one stopping criterion
Software Reliability Engineering can help also to determine when to stop testingby taking into consideration aspects like failure intensity
11
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 12110
12 What is testing - Defini tion of testing
12
Testing = the process concerned with planning the necessary static and dynamic
activities preparation and evaluation of software products and related deliverablesin order tobull determine that they satisfy specified requirementsbull demonstrate that they are fit for the intended usebull detect defects help and motivate the developers to fix thembull measure assess and improve the quality of the software product
Testing should be performed throughout the whole software life cycle
There are two basic types of testing execution and non-execution based
Other definitions
bull(IEEE) Testing = the process of analyzing a software item to detect the differencesbetween existing and required conditions and to evaluate its featuresbull(Myers (b3)) Testing = the process of executing a program with the intent of findingerrorsbull(Craig amp Jaskiel (b5)) Testing = a concurrent lifecycle process of engineering usingand maintaining test-ware in order to measure and improve the quality of thesoftware being tested
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 13110
12 What is testing ndash Testing ldquo schoolsrdquo
13
Analytic School - testing is rigorous academic and technicalbullTesting is a branch of CSMathematicsbullTesting techniques must have a logic-mathematical formbullKey Question Which techniques should we usebullRequire precise and detailed specifications
Factory School - testing is a way to measure progress with emphasis on cost and
repeatable standardsbullTesting must be managed amp cost effectivebullTesting validates the product amp measures development progressbullKey Questions How can we measure whether wersquore making progress When will we be donebullRequire clear boundaries between testing and other activities (startstop criteria)
bullEncourage standards (V-model) ldquobest practicesrdquo and certificationQuality School - emphasizes process amp quality acting as the gatekeeper bullSoftware quality requires disciplinebullTesters may need to police developers to follow the rulesbullKey Question Are we following a good process
bullTesting is a stepping stone to ldquoprocess improvementrdquoContext-Driven School - emphasizes people setting out to find the bugs that will bemost important to stakeholders
bullSoftware is created by people People set the contextbullTesting finds bugs acting as a skilled mental activitybullKey Question What testing would be most valuable right nowbullExpect changes Adapt testing plans based on test resultsbullTesting research requires empirical and psychological study
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 14110
13 General testing principles
14
bull Testing shows presence of defects but cannot prove that there are no moredefects testing can only reduce the probability of undiscovered defects
bull Complete exhaustive testing is impossible good strategy and risk managementmust be used
bull Pareto rule (defect clustering) usually 20 of the modules contain 80 of thebugs
bull Early testing testing activities should start as soon as possible (including hereplanning design reviews)
bull Pesticide paradox if the same set of tests are repeated over again no new bugswill be found the test cases should be reviewed modified and new test casesdeveloped
bull Context dependence test design and execution is context dependent (desktopweb applications real-time hellip)
bull Verification and Validation discovering defects cannot help a product that is not fitto the users needs
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 15110
13 General testing principles ndash heuristics of software testing
bullOperability - The better it works the more efficiently it can be tested
bullObservability - What we see is what we test
bull Controllability - The better we control the software the more the testing processcan be automated and optimized
bull Decomposability - By controlling the scope of testing we can quickly isolateproblems and perform effective and efficient testing
bull Simplicity - The less there is to test the more quickly we can test it
bull Stability - The fewer the changes the fewer are the disruptions to testing
bull Understandability - The more information we will have the smarter we will test
bull Suitability - The more we know about the intended use of the software the
better we can organize our testing to find important bugs15
1 4 1 F d t l t t h
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 16110
141 Fundamental test process ndash phases
-Test Planning amp Test control
-Test Analysis amp Design
-Test Implementation ampExecution
-Evaluating exit criteria ampReporting
-Test Closure activities
16
1 4 1 Fundamental test process planning amp control
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 17110
141 Fundamental test process ndash planning amp control
Planning1 Determine scope
bull Study project documents used software life-cycle specifications product desiredquality attributesbull Clarify test process expectations
2 Determine risksbull Choose quality risk analysis method (eg FMEA)
bull Document the list of risks probability impact priority identify mitigation actions3 Estimate testing effort determine costs develop schedule
bull Define necessary rolesbull Decompose test project into phases and tasks (WBS)bull Schedule tasks assign resources set-up dependencies
4 Refine planbull Select test strategy (how to do it what test types at which test levels)bull Select metrics to be used for defect tracking coverage monitoringbull Define entry and exit criteria
Controlbull Measure and analyze resultsbull Monitor testing progress coverage exit criteriabull Assign or reallocate resources update the test plan schedulebull Initiate corrective actionsbull Make decisions
17
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 18110
142 Fundamental test process ndash analysis amp design
bull Reviewing the test basis (such as requirements architecture designinterfaces)
bull Identifying test conditions or test requirements and required test data
based on analysis of test items and the specificationbull Designing the testsbull Choose test techniquesbull Identify test scenarios pre-conditions expected results post-
conditionsbull Identify possible test Oracles
bull Evaluating testability of the requirements and systembull Designing the test environment set-up and identifying any required
infrastructure and tools
(see Lee Copeland (b2))
18
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 19110
142 Fundamental test process ndash what is a test Oracle
19
bull The expected result (test outcome) must be defined at the test analysis stagebull Who will decide that (expected result = actual result) when the test will be
executed The Test Oracle
Test Oracle = A source to determine expected result a principle or mechanism torecognize a problem The Test Oracle can bebull an existing system (the old versionhellip)bull a document (specification user manual)bull a competent client representative
hellip but never the source code itself Oracles in use = Simplification of Risk do not assess lsquopass - failrsquo but instead
lsquoproblem - no problemrsquo
Problem Oracles and Automation - Our ability to automate testing isfundamentally constrained by our ability to create and use oracles
Possible issues
bull false alarmsbull missed bugs
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 20110
143 Fundamental test process ndash implementation amp execution
Test implementation
bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)
bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission
(see Rex Black (b4))
20
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 21110
143 Fundamental test process ndash priori tizing the Test Cases
Why prioritize the Test Cases
bullIt is not possible to test everything we must do our best in the time available
bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence
bullThis means we must prioritise and focus testing on the priorities
What to watch
bullSeverity of possible defectsbullProbability of possible defects
bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity
21
144 Fundamental test process ndash evaluating exit criteria and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 22110
144 Fundamental test process evaluating exit criteria andreporting
Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed
bull Check if testing mission should be changed
Test report ing
bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity
(see Rex Black (b4) amp RUP- Test discipline(s5))
22
1 4 5 F d l l
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 23110
145 Fundamental test process ndash test closure
bull Verify if test deliverables have been delivered
bull Check and close the remaining active bug reports
bull Archiving the test-ware and environment
bull Handover of the test environment
bull Analyze the identified test process problems (lessons learned)
bull Implement action plan based improvements
(see Rex Black (b4))
23
1 5 Th h l f i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 24110
15 The psychology of testing
24
Testing is regarded as a lsquodestructiversquo activity
(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts
bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise
bullShould be a planned organised kind ofperson
Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices
Rex BlackrsquosTop 10 professional errors
bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations
(see also Brian Marickrsquos article )
1 5 Th h l g f t ti g
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 25110
15 The psychology of testing
25
ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)
ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects
bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs
Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)
Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts
bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence
2 1 1 The V testing model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 26110
211 The V testing model
26
211 The V testing model ndash Verification amp Validationifi i C fi i b
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 27110
27
Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled
Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled
Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level
2 1 1 The W testing model ndash dynamic testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 28110
211 The W testing model ndash dynamic testing
28
2 1 1 The W testing model ndash static testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 29110
211 The W testing model static testing
29
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 30110
212 Software development models Waterfall
30
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 31110
212 Software development models Waterfall
Waterfall weaknesses
middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule
middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover
middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen
middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to
partition the system for delivery of pieces of the system
31
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 32110
p p yp
32
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 33110
p p yp
Rapid Prototype Model weaknesses
middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked
middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products
middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations
middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary
middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study
33
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 34110
34
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 35110
35
Incremental Model weaknesses
middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments
middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-
defined interfaces are required
middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 36110
36
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 37110
Spiral Model weaknesses
middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to
proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk
analysis and prototyping may be excessive
37
212 Software development models - Rational Unified Process
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 38110
38
213 Software development models ndash Testing life cycle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 39110
bull For each software activity there must be a corresponding testing activity
bull The objectives of the testing are specific to that ldquotestedrdquo activity
bull Plan analysis and design of a testing activity should be done during thecorresponding development activity
bull Review and inspection must be considered as part of testing activities
39
221 Test levels ndash Component testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 8110
114 Why is testing necessary ndash What is quality
8
Quality (ISO) = The totality of the characteristics of an entity that bear on its
ability to satisfy stated or implied needs
bull there are many more definitions
Testing means not only to Verify (the thing is done right)hellip but also to Validate (the right thing is done)
Software quality includes reliability usability efficiency maintainability andportabilityRELIABILITY The ability of the software product to perform its requiredfunctions under stated conditions for a specified period of time or for aspecified number of operationsUSABILITY The capability of the software to be understood learned used
and attractive to the user when used under specified conditionsEFFICIENCY The capability of the software product to provide appropriateperformance relative to the amount of resources used under stated conditionsMAINTAINABILITY The ease with which a software product can be modifiedto correct defects modified to meet new requirements modified to make futuremaintenance easier or adapted to a changed environmentPORTABILITY The ease with which the software product can be transferredfrom one hardware or software environment to another
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 9110
114 Why is testing necessary - Testing and quality
Testing does not inject Quality
into the product but will measurethe level of the Quality of theproduct
Quality can be measured forbull progressbull variance (planned versus actual)
Measuring product quality bullFunctional compliance - functional software requirements testing
bullNon functional requirementsbullTest coverage criteriabullDefect count or defect trend criteria
9
1 1 4 Wh i i li ib
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 10110
114 Why is testing necessary ndash quality attributes
The QUINT model (extended ISO model)
10
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 11110
115 Why is testing necessary - How much testing is enough
when to stop testingThe five basic criteria often used to decide are
bull Previously defined coverage goals have been metbull The defect discovery rate has dropped below a previously defined thresholdbull The cost of finding the next defect exceeds the expected loss from that defectbull The project team reaches consensus that it is appropriate to release the productbull The manager decides to deliver the product
All these criteria are risk basedIt is important not depending on only one stopping criterion
Software Reliability Engineering can help also to determine when to stop testingby taking into consideration aspects like failure intensity
11
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 12110
12 What is testing - Defini tion of testing
12
Testing = the process concerned with planning the necessary static and dynamic
activities preparation and evaluation of software products and related deliverablesin order tobull determine that they satisfy specified requirementsbull demonstrate that they are fit for the intended usebull detect defects help and motivate the developers to fix thembull measure assess and improve the quality of the software product
Testing should be performed throughout the whole software life cycle
There are two basic types of testing execution and non-execution based
Other definitions
bull(IEEE) Testing = the process of analyzing a software item to detect the differencesbetween existing and required conditions and to evaluate its featuresbull(Myers (b3)) Testing = the process of executing a program with the intent of findingerrorsbull(Craig amp Jaskiel (b5)) Testing = a concurrent lifecycle process of engineering usingand maintaining test-ware in order to measure and improve the quality of thesoftware being tested
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 13110
12 What is testing ndash Testing ldquo schoolsrdquo
13
Analytic School - testing is rigorous academic and technicalbullTesting is a branch of CSMathematicsbullTesting techniques must have a logic-mathematical formbullKey Question Which techniques should we usebullRequire precise and detailed specifications
Factory School - testing is a way to measure progress with emphasis on cost and
repeatable standardsbullTesting must be managed amp cost effectivebullTesting validates the product amp measures development progressbullKey Questions How can we measure whether wersquore making progress When will we be donebullRequire clear boundaries between testing and other activities (startstop criteria)
bullEncourage standards (V-model) ldquobest practicesrdquo and certificationQuality School - emphasizes process amp quality acting as the gatekeeper bullSoftware quality requires disciplinebullTesters may need to police developers to follow the rulesbullKey Question Are we following a good process
bullTesting is a stepping stone to ldquoprocess improvementrdquoContext-Driven School - emphasizes people setting out to find the bugs that will bemost important to stakeholders
bullSoftware is created by people People set the contextbullTesting finds bugs acting as a skilled mental activitybullKey Question What testing would be most valuable right nowbullExpect changes Adapt testing plans based on test resultsbullTesting research requires empirical and psychological study
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 14110
13 General testing principles
14
bull Testing shows presence of defects but cannot prove that there are no moredefects testing can only reduce the probability of undiscovered defects
bull Complete exhaustive testing is impossible good strategy and risk managementmust be used
bull Pareto rule (defect clustering) usually 20 of the modules contain 80 of thebugs
bull Early testing testing activities should start as soon as possible (including hereplanning design reviews)
bull Pesticide paradox if the same set of tests are repeated over again no new bugswill be found the test cases should be reviewed modified and new test casesdeveloped
bull Context dependence test design and execution is context dependent (desktopweb applications real-time hellip)
bull Verification and Validation discovering defects cannot help a product that is not fitto the users needs
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 15110
13 General testing principles ndash heuristics of software testing
bullOperability - The better it works the more efficiently it can be tested
bullObservability - What we see is what we test
bull Controllability - The better we control the software the more the testing processcan be automated and optimized
bull Decomposability - By controlling the scope of testing we can quickly isolateproblems and perform effective and efficient testing
bull Simplicity - The less there is to test the more quickly we can test it
bull Stability - The fewer the changes the fewer are the disruptions to testing
bull Understandability - The more information we will have the smarter we will test
bull Suitability - The more we know about the intended use of the software the
better we can organize our testing to find important bugs15
1 4 1 F d t l t t h
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 16110
141 Fundamental test process ndash phases
-Test Planning amp Test control
-Test Analysis amp Design
-Test Implementation ampExecution
-Evaluating exit criteria ampReporting
-Test Closure activities
16
1 4 1 Fundamental test process planning amp control
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 17110
141 Fundamental test process ndash planning amp control
Planning1 Determine scope
bull Study project documents used software life-cycle specifications product desiredquality attributesbull Clarify test process expectations
2 Determine risksbull Choose quality risk analysis method (eg FMEA)
bull Document the list of risks probability impact priority identify mitigation actions3 Estimate testing effort determine costs develop schedule
bull Define necessary rolesbull Decompose test project into phases and tasks (WBS)bull Schedule tasks assign resources set-up dependencies
4 Refine planbull Select test strategy (how to do it what test types at which test levels)bull Select metrics to be used for defect tracking coverage monitoringbull Define entry and exit criteria
Controlbull Measure and analyze resultsbull Monitor testing progress coverage exit criteriabull Assign or reallocate resources update the test plan schedulebull Initiate corrective actionsbull Make decisions
17
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 18110
142 Fundamental test process ndash analysis amp design
bull Reviewing the test basis (such as requirements architecture designinterfaces)
bull Identifying test conditions or test requirements and required test data
based on analysis of test items and the specificationbull Designing the testsbull Choose test techniquesbull Identify test scenarios pre-conditions expected results post-
conditionsbull Identify possible test Oracles
bull Evaluating testability of the requirements and systembull Designing the test environment set-up and identifying any required
infrastructure and tools
(see Lee Copeland (b2))
18
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 19110
142 Fundamental test process ndash what is a test Oracle
19
bull The expected result (test outcome) must be defined at the test analysis stagebull Who will decide that (expected result = actual result) when the test will be
executed The Test Oracle
Test Oracle = A source to determine expected result a principle or mechanism torecognize a problem The Test Oracle can bebull an existing system (the old versionhellip)bull a document (specification user manual)bull a competent client representative
hellip but never the source code itself Oracles in use = Simplification of Risk do not assess lsquopass - failrsquo but instead
lsquoproblem - no problemrsquo
Problem Oracles and Automation - Our ability to automate testing isfundamentally constrained by our ability to create and use oracles
Possible issues
bull false alarmsbull missed bugs
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 20110
143 Fundamental test process ndash implementation amp execution
Test implementation
bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)
bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission
(see Rex Black (b4))
20
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 21110
143 Fundamental test process ndash priori tizing the Test Cases
Why prioritize the Test Cases
bullIt is not possible to test everything we must do our best in the time available
bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence
bullThis means we must prioritise and focus testing on the priorities
What to watch
bullSeverity of possible defectsbullProbability of possible defects
bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity
21
144 Fundamental test process ndash evaluating exit criteria and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 22110
144 Fundamental test process evaluating exit criteria andreporting
Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed
bull Check if testing mission should be changed
Test report ing
bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity
(see Rex Black (b4) amp RUP- Test discipline(s5))
22
1 4 5 F d l l
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 23110
145 Fundamental test process ndash test closure
bull Verify if test deliverables have been delivered
bull Check and close the remaining active bug reports
bull Archiving the test-ware and environment
bull Handover of the test environment
bull Analyze the identified test process problems (lessons learned)
bull Implement action plan based improvements
(see Rex Black (b4))
23
1 5 Th h l f i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 24110
15 The psychology of testing
24
Testing is regarded as a lsquodestructiversquo activity
(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts
bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise
bullShould be a planned organised kind ofperson
Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices
Rex BlackrsquosTop 10 professional errors
bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations
(see also Brian Marickrsquos article )
1 5 Th h l g f t ti g
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 25110
15 The psychology of testing
25
ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)
ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects
bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs
Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)
Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts
bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence
2 1 1 The V testing model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 26110
211 The V testing model
26
211 The V testing model ndash Verification amp Validationifi i C fi i b
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 27110
27
Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled
Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled
Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level
2 1 1 The W testing model ndash dynamic testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 28110
211 The W testing model ndash dynamic testing
28
2 1 1 The W testing model ndash static testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 29110
211 The W testing model static testing
29
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 30110
212 Software development models Waterfall
30
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 31110
212 Software development models Waterfall
Waterfall weaknesses
middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule
middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover
middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen
middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to
partition the system for delivery of pieces of the system
31
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 32110
p p yp
32
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 33110
p p yp
Rapid Prototype Model weaknesses
middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked
middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products
middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations
middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary
middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study
33
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 34110
34
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 35110
35
Incremental Model weaknesses
middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments
middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-
defined interfaces are required
middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 36110
36
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 37110
Spiral Model weaknesses
middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to
proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk
analysis and prototyping may be excessive
37
212 Software development models - Rational Unified Process
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 38110
38
213 Software development models ndash Testing life cycle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 39110
bull For each software activity there must be a corresponding testing activity
bull The objectives of the testing are specific to that ldquotestedrdquo activity
bull Plan analysis and design of a testing activity should be done during thecorresponding development activity
bull Review and inspection must be considered as part of testing activities
39
221 Test levels ndash Component testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 9110
114 Why is testing necessary - Testing and quality
Testing does not inject Quality
into the product but will measurethe level of the Quality of theproduct
Quality can be measured forbull progressbull variance (planned versus actual)
Measuring product quality bullFunctional compliance - functional software requirements testing
bullNon functional requirementsbullTest coverage criteriabullDefect count or defect trend criteria
9
1 1 4 Wh i i li ib
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 10110
114 Why is testing necessary ndash quality attributes
The QUINT model (extended ISO model)
10
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 11110
115 Why is testing necessary - How much testing is enough
when to stop testingThe five basic criteria often used to decide are
bull Previously defined coverage goals have been metbull The defect discovery rate has dropped below a previously defined thresholdbull The cost of finding the next defect exceeds the expected loss from that defectbull The project team reaches consensus that it is appropriate to release the productbull The manager decides to deliver the product
All these criteria are risk basedIt is important not depending on only one stopping criterion
Software Reliability Engineering can help also to determine when to stop testingby taking into consideration aspects like failure intensity
11
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 12110
12 What is testing - Defini tion of testing
12
Testing = the process concerned with planning the necessary static and dynamic
activities preparation and evaluation of software products and related deliverablesin order tobull determine that they satisfy specified requirementsbull demonstrate that they are fit for the intended usebull detect defects help and motivate the developers to fix thembull measure assess and improve the quality of the software product
Testing should be performed throughout the whole software life cycle
There are two basic types of testing execution and non-execution based
Other definitions
bull(IEEE) Testing = the process of analyzing a software item to detect the differencesbetween existing and required conditions and to evaluate its featuresbull(Myers (b3)) Testing = the process of executing a program with the intent of findingerrorsbull(Craig amp Jaskiel (b5)) Testing = a concurrent lifecycle process of engineering usingand maintaining test-ware in order to measure and improve the quality of thesoftware being tested
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 13110
12 What is testing ndash Testing ldquo schoolsrdquo
13
Analytic School - testing is rigorous academic and technicalbullTesting is a branch of CSMathematicsbullTesting techniques must have a logic-mathematical formbullKey Question Which techniques should we usebullRequire precise and detailed specifications
Factory School - testing is a way to measure progress with emphasis on cost and
repeatable standardsbullTesting must be managed amp cost effectivebullTesting validates the product amp measures development progressbullKey Questions How can we measure whether wersquore making progress When will we be donebullRequire clear boundaries between testing and other activities (startstop criteria)
bullEncourage standards (V-model) ldquobest practicesrdquo and certificationQuality School - emphasizes process amp quality acting as the gatekeeper bullSoftware quality requires disciplinebullTesters may need to police developers to follow the rulesbullKey Question Are we following a good process
bullTesting is a stepping stone to ldquoprocess improvementrdquoContext-Driven School - emphasizes people setting out to find the bugs that will bemost important to stakeholders
bullSoftware is created by people People set the contextbullTesting finds bugs acting as a skilled mental activitybullKey Question What testing would be most valuable right nowbullExpect changes Adapt testing plans based on test resultsbullTesting research requires empirical and psychological study
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 14110
13 General testing principles
14
bull Testing shows presence of defects but cannot prove that there are no moredefects testing can only reduce the probability of undiscovered defects
bull Complete exhaustive testing is impossible good strategy and risk managementmust be used
bull Pareto rule (defect clustering) usually 20 of the modules contain 80 of thebugs
bull Early testing testing activities should start as soon as possible (including hereplanning design reviews)
bull Pesticide paradox if the same set of tests are repeated over again no new bugswill be found the test cases should be reviewed modified and new test casesdeveloped
bull Context dependence test design and execution is context dependent (desktopweb applications real-time hellip)
bull Verification and Validation discovering defects cannot help a product that is not fitto the users needs
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 15110
13 General testing principles ndash heuristics of software testing
bullOperability - The better it works the more efficiently it can be tested
bullObservability - What we see is what we test
bull Controllability - The better we control the software the more the testing processcan be automated and optimized
bull Decomposability - By controlling the scope of testing we can quickly isolateproblems and perform effective and efficient testing
bull Simplicity - The less there is to test the more quickly we can test it
bull Stability - The fewer the changes the fewer are the disruptions to testing
bull Understandability - The more information we will have the smarter we will test
bull Suitability - The more we know about the intended use of the software the
better we can organize our testing to find important bugs15
1 4 1 F d t l t t h
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 16110
141 Fundamental test process ndash phases
-Test Planning amp Test control
-Test Analysis amp Design
-Test Implementation ampExecution
-Evaluating exit criteria ampReporting
-Test Closure activities
16
1 4 1 Fundamental test process planning amp control
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 17110
141 Fundamental test process ndash planning amp control
Planning1 Determine scope
bull Study project documents used software life-cycle specifications product desiredquality attributesbull Clarify test process expectations
2 Determine risksbull Choose quality risk analysis method (eg FMEA)
bull Document the list of risks probability impact priority identify mitigation actions3 Estimate testing effort determine costs develop schedule
bull Define necessary rolesbull Decompose test project into phases and tasks (WBS)bull Schedule tasks assign resources set-up dependencies
4 Refine planbull Select test strategy (how to do it what test types at which test levels)bull Select metrics to be used for defect tracking coverage monitoringbull Define entry and exit criteria
Controlbull Measure and analyze resultsbull Monitor testing progress coverage exit criteriabull Assign or reallocate resources update the test plan schedulebull Initiate corrective actionsbull Make decisions
17
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 18110
142 Fundamental test process ndash analysis amp design
bull Reviewing the test basis (such as requirements architecture designinterfaces)
bull Identifying test conditions or test requirements and required test data
based on analysis of test items and the specificationbull Designing the testsbull Choose test techniquesbull Identify test scenarios pre-conditions expected results post-
conditionsbull Identify possible test Oracles
bull Evaluating testability of the requirements and systembull Designing the test environment set-up and identifying any required
infrastructure and tools
(see Lee Copeland (b2))
18
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 19110
142 Fundamental test process ndash what is a test Oracle
19
bull The expected result (test outcome) must be defined at the test analysis stagebull Who will decide that (expected result = actual result) when the test will be
executed The Test Oracle
Test Oracle = A source to determine expected result a principle or mechanism torecognize a problem The Test Oracle can bebull an existing system (the old versionhellip)bull a document (specification user manual)bull a competent client representative
hellip but never the source code itself Oracles in use = Simplification of Risk do not assess lsquopass - failrsquo but instead
lsquoproblem - no problemrsquo
Problem Oracles and Automation - Our ability to automate testing isfundamentally constrained by our ability to create and use oracles
Possible issues
bull false alarmsbull missed bugs
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 20110
143 Fundamental test process ndash implementation amp execution
Test implementation
bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)
bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission
(see Rex Black (b4))
20
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 21110
143 Fundamental test process ndash priori tizing the Test Cases
Why prioritize the Test Cases
bullIt is not possible to test everything we must do our best in the time available
bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence
bullThis means we must prioritise and focus testing on the priorities
What to watch
bullSeverity of possible defectsbullProbability of possible defects
bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity
21
144 Fundamental test process ndash evaluating exit criteria and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 22110
144 Fundamental test process evaluating exit criteria andreporting
Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed
bull Check if testing mission should be changed
Test report ing
bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity
(see Rex Black (b4) amp RUP- Test discipline(s5))
22
1 4 5 F d l l
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 23110
145 Fundamental test process ndash test closure
bull Verify if test deliverables have been delivered
bull Check and close the remaining active bug reports
bull Archiving the test-ware and environment
bull Handover of the test environment
bull Analyze the identified test process problems (lessons learned)
bull Implement action plan based improvements
(see Rex Black (b4))
23
1 5 Th h l f i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 24110
15 The psychology of testing
24
Testing is regarded as a lsquodestructiversquo activity
(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts
bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise
bullShould be a planned organised kind ofperson
Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices
Rex BlackrsquosTop 10 professional errors
bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations
(see also Brian Marickrsquos article )
1 5 Th h l g f t ti g
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 25110
15 The psychology of testing
25
ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)
ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects
bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs
Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)
Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts
bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence
2 1 1 The V testing model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 26110
211 The V testing model
26
211 The V testing model ndash Verification amp Validationifi i C fi i b
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 27110
27
Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled
Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled
Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level
2 1 1 The W testing model ndash dynamic testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 28110
211 The W testing model ndash dynamic testing
28
2 1 1 The W testing model ndash static testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 29110
211 The W testing model static testing
29
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 30110
212 Software development models Waterfall
30
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 31110
212 Software development models Waterfall
Waterfall weaknesses
middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule
middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover
middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen
middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to
partition the system for delivery of pieces of the system
31
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 32110
p p yp
32
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 33110
p p yp
Rapid Prototype Model weaknesses
middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked
middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products
middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations
middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary
middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study
33
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 34110
34
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 35110
35
Incremental Model weaknesses
middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments
middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-
defined interfaces are required
middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 36110
36
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 37110
Spiral Model weaknesses
middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to
proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk
analysis and prototyping may be excessive
37
212 Software development models - Rational Unified Process
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 38110
38
213 Software development models ndash Testing life cycle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 39110
bull For each software activity there must be a corresponding testing activity
bull The objectives of the testing are specific to that ldquotestedrdquo activity
bull Plan analysis and design of a testing activity should be done during thecorresponding development activity
bull Review and inspection must be considered as part of testing activities
39
221 Test levels ndash Component testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 10110
114 Why is testing necessary ndash quality attributes
The QUINT model (extended ISO model)
10
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 11110
115 Why is testing necessary - How much testing is enough
when to stop testingThe five basic criteria often used to decide are
bull Previously defined coverage goals have been metbull The defect discovery rate has dropped below a previously defined thresholdbull The cost of finding the next defect exceeds the expected loss from that defectbull The project team reaches consensus that it is appropriate to release the productbull The manager decides to deliver the product
All these criteria are risk basedIt is important not depending on only one stopping criterion
Software Reliability Engineering can help also to determine when to stop testingby taking into consideration aspects like failure intensity
11
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 12110
12 What is testing - Defini tion of testing
12
Testing = the process concerned with planning the necessary static and dynamic
activities preparation and evaluation of software products and related deliverablesin order tobull determine that they satisfy specified requirementsbull demonstrate that they are fit for the intended usebull detect defects help and motivate the developers to fix thembull measure assess and improve the quality of the software product
Testing should be performed throughout the whole software life cycle
There are two basic types of testing execution and non-execution based
Other definitions
bull(IEEE) Testing = the process of analyzing a software item to detect the differencesbetween existing and required conditions and to evaluate its featuresbull(Myers (b3)) Testing = the process of executing a program with the intent of findingerrorsbull(Craig amp Jaskiel (b5)) Testing = a concurrent lifecycle process of engineering usingand maintaining test-ware in order to measure and improve the quality of thesoftware being tested
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 13110
12 What is testing ndash Testing ldquo schoolsrdquo
13
Analytic School - testing is rigorous academic and technicalbullTesting is a branch of CSMathematicsbullTesting techniques must have a logic-mathematical formbullKey Question Which techniques should we usebullRequire precise and detailed specifications
Factory School - testing is a way to measure progress with emphasis on cost and
repeatable standardsbullTesting must be managed amp cost effectivebullTesting validates the product amp measures development progressbullKey Questions How can we measure whether wersquore making progress When will we be donebullRequire clear boundaries between testing and other activities (startstop criteria)
bullEncourage standards (V-model) ldquobest practicesrdquo and certificationQuality School - emphasizes process amp quality acting as the gatekeeper bullSoftware quality requires disciplinebullTesters may need to police developers to follow the rulesbullKey Question Are we following a good process
bullTesting is a stepping stone to ldquoprocess improvementrdquoContext-Driven School - emphasizes people setting out to find the bugs that will bemost important to stakeholders
bullSoftware is created by people People set the contextbullTesting finds bugs acting as a skilled mental activitybullKey Question What testing would be most valuable right nowbullExpect changes Adapt testing plans based on test resultsbullTesting research requires empirical and psychological study
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 14110
13 General testing principles
14
bull Testing shows presence of defects but cannot prove that there are no moredefects testing can only reduce the probability of undiscovered defects
bull Complete exhaustive testing is impossible good strategy and risk managementmust be used
bull Pareto rule (defect clustering) usually 20 of the modules contain 80 of thebugs
bull Early testing testing activities should start as soon as possible (including hereplanning design reviews)
bull Pesticide paradox if the same set of tests are repeated over again no new bugswill be found the test cases should be reviewed modified and new test casesdeveloped
bull Context dependence test design and execution is context dependent (desktopweb applications real-time hellip)
bull Verification and Validation discovering defects cannot help a product that is not fitto the users needs
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 15110
13 General testing principles ndash heuristics of software testing
bullOperability - The better it works the more efficiently it can be tested
bullObservability - What we see is what we test
bull Controllability - The better we control the software the more the testing processcan be automated and optimized
bull Decomposability - By controlling the scope of testing we can quickly isolateproblems and perform effective and efficient testing
bull Simplicity - The less there is to test the more quickly we can test it
bull Stability - The fewer the changes the fewer are the disruptions to testing
bull Understandability - The more information we will have the smarter we will test
bull Suitability - The more we know about the intended use of the software the
better we can organize our testing to find important bugs15
1 4 1 F d t l t t h
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 16110
141 Fundamental test process ndash phases
-Test Planning amp Test control
-Test Analysis amp Design
-Test Implementation ampExecution
-Evaluating exit criteria ampReporting
-Test Closure activities
16
1 4 1 Fundamental test process planning amp control
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 17110
141 Fundamental test process ndash planning amp control
Planning1 Determine scope
bull Study project documents used software life-cycle specifications product desiredquality attributesbull Clarify test process expectations
2 Determine risksbull Choose quality risk analysis method (eg FMEA)
bull Document the list of risks probability impact priority identify mitigation actions3 Estimate testing effort determine costs develop schedule
bull Define necessary rolesbull Decompose test project into phases and tasks (WBS)bull Schedule tasks assign resources set-up dependencies
4 Refine planbull Select test strategy (how to do it what test types at which test levels)bull Select metrics to be used for defect tracking coverage monitoringbull Define entry and exit criteria
Controlbull Measure and analyze resultsbull Monitor testing progress coverage exit criteriabull Assign or reallocate resources update the test plan schedulebull Initiate corrective actionsbull Make decisions
17
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 18110
142 Fundamental test process ndash analysis amp design
bull Reviewing the test basis (such as requirements architecture designinterfaces)
bull Identifying test conditions or test requirements and required test data
based on analysis of test items and the specificationbull Designing the testsbull Choose test techniquesbull Identify test scenarios pre-conditions expected results post-
conditionsbull Identify possible test Oracles
bull Evaluating testability of the requirements and systembull Designing the test environment set-up and identifying any required
infrastructure and tools
(see Lee Copeland (b2))
18
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 19110
142 Fundamental test process ndash what is a test Oracle
19
bull The expected result (test outcome) must be defined at the test analysis stagebull Who will decide that (expected result = actual result) when the test will be
executed The Test Oracle
Test Oracle = A source to determine expected result a principle or mechanism torecognize a problem The Test Oracle can bebull an existing system (the old versionhellip)bull a document (specification user manual)bull a competent client representative
hellip but never the source code itself Oracles in use = Simplification of Risk do not assess lsquopass - failrsquo but instead
lsquoproblem - no problemrsquo
Problem Oracles and Automation - Our ability to automate testing isfundamentally constrained by our ability to create and use oracles
Possible issues
bull false alarmsbull missed bugs
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 20110
143 Fundamental test process ndash implementation amp execution
Test implementation
bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)
bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission
(see Rex Black (b4))
20
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 21110
143 Fundamental test process ndash priori tizing the Test Cases
Why prioritize the Test Cases
bullIt is not possible to test everything we must do our best in the time available
bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence
bullThis means we must prioritise and focus testing on the priorities
What to watch
bullSeverity of possible defectsbullProbability of possible defects
bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity
21
144 Fundamental test process ndash evaluating exit criteria and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 22110
144 Fundamental test process evaluating exit criteria andreporting
Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed
bull Check if testing mission should be changed
Test report ing
bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity
(see Rex Black (b4) amp RUP- Test discipline(s5))
22
1 4 5 F d l l
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 23110
145 Fundamental test process ndash test closure
bull Verify if test deliverables have been delivered
bull Check and close the remaining active bug reports
bull Archiving the test-ware and environment
bull Handover of the test environment
bull Analyze the identified test process problems (lessons learned)
bull Implement action plan based improvements
(see Rex Black (b4))
23
1 5 Th h l f i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 24110
15 The psychology of testing
24
Testing is regarded as a lsquodestructiversquo activity
(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts
bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise
bullShould be a planned organised kind ofperson
Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices
Rex BlackrsquosTop 10 professional errors
bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations
(see also Brian Marickrsquos article )
1 5 Th h l g f t ti g
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 25110
15 The psychology of testing
25
ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)
ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects
bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs
Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)
Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts
bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence
2 1 1 The V testing model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 26110
211 The V testing model
26
211 The V testing model ndash Verification amp Validationifi i C fi i b
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 27110
27
Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled
Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled
Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level
2 1 1 The W testing model ndash dynamic testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 28110
211 The W testing model ndash dynamic testing
28
2 1 1 The W testing model ndash static testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 29110
211 The W testing model static testing
29
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 30110
212 Software development models Waterfall
30
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 31110
212 Software development models Waterfall
Waterfall weaknesses
middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule
middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover
middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen
middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to
partition the system for delivery of pieces of the system
31
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 32110
p p yp
32
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 33110
p p yp
Rapid Prototype Model weaknesses
middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked
middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products
middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations
middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary
middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study
33
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 34110
34
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 35110
35
Incremental Model weaknesses
middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments
middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-
defined interfaces are required
middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 36110
36
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 37110
Spiral Model weaknesses
middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to
proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk
analysis and prototyping may be excessive
37
212 Software development models - Rational Unified Process
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 38110
38
213 Software development models ndash Testing life cycle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 39110
bull For each software activity there must be a corresponding testing activity
bull The objectives of the testing are specific to that ldquotestedrdquo activity
bull Plan analysis and design of a testing activity should be done during thecorresponding development activity
bull Review and inspection must be considered as part of testing activities
39
221 Test levels ndash Component testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 11110
115 Why is testing necessary - How much testing is enough
when to stop testingThe five basic criteria often used to decide are
bull Previously defined coverage goals have been metbull The defect discovery rate has dropped below a previously defined thresholdbull The cost of finding the next defect exceeds the expected loss from that defectbull The project team reaches consensus that it is appropriate to release the productbull The manager decides to deliver the product
All these criteria are risk basedIt is important not depending on only one stopping criterion
Software Reliability Engineering can help also to determine when to stop testingby taking into consideration aspects like failure intensity
11
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 12110
12 What is testing - Defini tion of testing
12
Testing = the process concerned with planning the necessary static and dynamic
activities preparation and evaluation of software products and related deliverablesin order tobull determine that they satisfy specified requirementsbull demonstrate that they are fit for the intended usebull detect defects help and motivate the developers to fix thembull measure assess and improve the quality of the software product
Testing should be performed throughout the whole software life cycle
There are two basic types of testing execution and non-execution based
Other definitions
bull(IEEE) Testing = the process of analyzing a software item to detect the differencesbetween existing and required conditions and to evaluate its featuresbull(Myers (b3)) Testing = the process of executing a program with the intent of findingerrorsbull(Craig amp Jaskiel (b5)) Testing = a concurrent lifecycle process of engineering usingand maintaining test-ware in order to measure and improve the quality of thesoftware being tested
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 13110
12 What is testing ndash Testing ldquo schoolsrdquo
13
Analytic School - testing is rigorous academic and technicalbullTesting is a branch of CSMathematicsbullTesting techniques must have a logic-mathematical formbullKey Question Which techniques should we usebullRequire precise and detailed specifications
Factory School - testing is a way to measure progress with emphasis on cost and
repeatable standardsbullTesting must be managed amp cost effectivebullTesting validates the product amp measures development progressbullKey Questions How can we measure whether wersquore making progress When will we be donebullRequire clear boundaries between testing and other activities (startstop criteria)
bullEncourage standards (V-model) ldquobest practicesrdquo and certificationQuality School - emphasizes process amp quality acting as the gatekeeper bullSoftware quality requires disciplinebullTesters may need to police developers to follow the rulesbullKey Question Are we following a good process
bullTesting is a stepping stone to ldquoprocess improvementrdquoContext-Driven School - emphasizes people setting out to find the bugs that will bemost important to stakeholders
bullSoftware is created by people People set the contextbullTesting finds bugs acting as a skilled mental activitybullKey Question What testing would be most valuable right nowbullExpect changes Adapt testing plans based on test resultsbullTesting research requires empirical and psychological study
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 14110
13 General testing principles
14
bull Testing shows presence of defects but cannot prove that there are no moredefects testing can only reduce the probability of undiscovered defects
bull Complete exhaustive testing is impossible good strategy and risk managementmust be used
bull Pareto rule (defect clustering) usually 20 of the modules contain 80 of thebugs
bull Early testing testing activities should start as soon as possible (including hereplanning design reviews)
bull Pesticide paradox if the same set of tests are repeated over again no new bugswill be found the test cases should be reviewed modified and new test casesdeveloped
bull Context dependence test design and execution is context dependent (desktopweb applications real-time hellip)
bull Verification and Validation discovering defects cannot help a product that is not fitto the users needs
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 15110
13 General testing principles ndash heuristics of software testing
bullOperability - The better it works the more efficiently it can be tested
bullObservability - What we see is what we test
bull Controllability - The better we control the software the more the testing processcan be automated and optimized
bull Decomposability - By controlling the scope of testing we can quickly isolateproblems and perform effective and efficient testing
bull Simplicity - The less there is to test the more quickly we can test it
bull Stability - The fewer the changes the fewer are the disruptions to testing
bull Understandability - The more information we will have the smarter we will test
bull Suitability - The more we know about the intended use of the software the
better we can organize our testing to find important bugs15
1 4 1 F d t l t t h
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 16110
141 Fundamental test process ndash phases
-Test Planning amp Test control
-Test Analysis amp Design
-Test Implementation ampExecution
-Evaluating exit criteria ampReporting
-Test Closure activities
16
1 4 1 Fundamental test process planning amp control
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 17110
141 Fundamental test process ndash planning amp control
Planning1 Determine scope
bull Study project documents used software life-cycle specifications product desiredquality attributesbull Clarify test process expectations
2 Determine risksbull Choose quality risk analysis method (eg FMEA)
bull Document the list of risks probability impact priority identify mitigation actions3 Estimate testing effort determine costs develop schedule
bull Define necessary rolesbull Decompose test project into phases and tasks (WBS)bull Schedule tasks assign resources set-up dependencies
4 Refine planbull Select test strategy (how to do it what test types at which test levels)bull Select metrics to be used for defect tracking coverage monitoringbull Define entry and exit criteria
Controlbull Measure and analyze resultsbull Monitor testing progress coverage exit criteriabull Assign or reallocate resources update the test plan schedulebull Initiate corrective actionsbull Make decisions
17
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 18110
142 Fundamental test process ndash analysis amp design
bull Reviewing the test basis (such as requirements architecture designinterfaces)
bull Identifying test conditions or test requirements and required test data
based on analysis of test items and the specificationbull Designing the testsbull Choose test techniquesbull Identify test scenarios pre-conditions expected results post-
conditionsbull Identify possible test Oracles
bull Evaluating testability of the requirements and systembull Designing the test environment set-up and identifying any required
infrastructure and tools
(see Lee Copeland (b2))
18
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 19110
142 Fundamental test process ndash what is a test Oracle
19
bull The expected result (test outcome) must be defined at the test analysis stagebull Who will decide that (expected result = actual result) when the test will be
executed The Test Oracle
Test Oracle = A source to determine expected result a principle or mechanism torecognize a problem The Test Oracle can bebull an existing system (the old versionhellip)bull a document (specification user manual)bull a competent client representative
hellip but never the source code itself Oracles in use = Simplification of Risk do not assess lsquopass - failrsquo but instead
lsquoproblem - no problemrsquo
Problem Oracles and Automation - Our ability to automate testing isfundamentally constrained by our ability to create and use oracles
Possible issues
bull false alarmsbull missed bugs
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 20110
143 Fundamental test process ndash implementation amp execution
Test implementation
bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)
bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission
(see Rex Black (b4))
20
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 21110
143 Fundamental test process ndash priori tizing the Test Cases
Why prioritize the Test Cases
bullIt is not possible to test everything we must do our best in the time available
bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence
bullThis means we must prioritise and focus testing on the priorities
What to watch
bullSeverity of possible defectsbullProbability of possible defects
bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity
21
144 Fundamental test process ndash evaluating exit criteria and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 22110
144 Fundamental test process evaluating exit criteria andreporting
Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed
bull Check if testing mission should be changed
Test report ing
bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity
(see Rex Black (b4) amp RUP- Test discipline(s5))
22
1 4 5 F d l l
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 23110
145 Fundamental test process ndash test closure
bull Verify if test deliverables have been delivered
bull Check and close the remaining active bug reports
bull Archiving the test-ware and environment
bull Handover of the test environment
bull Analyze the identified test process problems (lessons learned)
bull Implement action plan based improvements
(see Rex Black (b4))
23
1 5 Th h l f i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 24110
15 The psychology of testing
24
Testing is regarded as a lsquodestructiversquo activity
(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts
bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise
bullShould be a planned organised kind ofperson
Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices
Rex BlackrsquosTop 10 professional errors
bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations
(see also Brian Marickrsquos article )
1 5 Th h l g f t ti g
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 25110
15 The psychology of testing
25
ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)
ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects
bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs
Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)
Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts
bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence
2 1 1 The V testing model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 26110
211 The V testing model
26
211 The V testing model ndash Verification amp Validationifi i C fi i b
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 27110
27
Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled
Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled
Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level
2 1 1 The W testing model ndash dynamic testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 28110
211 The W testing model ndash dynamic testing
28
2 1 1 The W testing model ndash static testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 29110
211 The W testing model static testing
29
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 30110
212 Software development models Waterfall
30
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 31110
212 Software development models Waterfall
Waterfall weaknesses
middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule
middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover
middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen
middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to
partition the system for delivery of pieces of the system
31
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 32110
p p yp
32
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 33110
p p yp
Rapid Prototype Model weaknesses
middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked
middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products
middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations
middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary
middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study
33
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 34110
34
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 35110
35
Incremental Model weaknesses
middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments
middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-
defined interfaces are required
middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 36110
36
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 37110
Spiral Model weaknesses
middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to
proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk
analysis and prototyping may be excessive
37
212 Software development models - Rational Unified Process
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 38110
38
213 Software development models ndash Testing life cycle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 39110
bull For each software activity there must be a corresponding testing activity
bull The objectives of the testing are specific to that ldquotestedrdquo activity
bull Plan analysis and design of a testing activity should be done during thecorresponding development activity
bull Review and inspection must be considered as part of testing activities
39
221 Test levels ndash Component testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 12110
12 What is testing - Defini tion of testing
12
Testing = the process concerned with planning the necessary static and dynamic
activities preparation and evaluation of software products and related deliverablesin order tobull determine that they satisfy specified requirementsbull demonstrate that they are fit for the intended usebull detect defects help and motivate the developers to fix thembull measure assess and improve the quality of the software product
Testing should be performed throughout the whole software life cycle
There are two basic types of testing execution and non-execution based
Other definitions
bull(IEEE) Testing = the process of analyzing a software item to detect the differencesbetween existing and required conditions and to evaluate its featuresbull(Myers (b3)) Testing = the process of executing a program with the intent of findingerrorsbull(Craig amp Jaskiel (b5)) Testing = a concurrent lifecycle process of engineering usingand maintaining test-ware in order to measure and improve the quality of thesoftware being tested
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 13110
12 What is testing ndash Testing ldquo schoolsrdquo
13
Analytic School - testing is rigorous academic and technicalbullTesting is a branch of CSMathematicsbullTesting techniques must have a logic-mathematical formbullKey Question Which techniques should we usebullRequire precise and detailed specifications
Factory School - testing is a way to measure progress with emphasis on cost and
repeatable standardsbullTesting must be managed amp cost effectivebullTesting validates the product amp measures development progressbullKey Questions How can we measure whether wersquore making progress When will we be donebullRequire clear boundaries between testing and other activities (startstop criteria)
bullEncourage standards (V-model) ldquobest practicesrdquo and certificationQuality School - emphasizes process amp quality acting as the gatekeeper bullSoftware quality requires disciplinebullTesters may need to police developers to follow the rulesbullKey Question Are we following a good process
bullTesting is a stepping stone to ldquoprocess improvementrdquoContext-Driven School - emphasizes people setting out to find the bugs that will bemost important to stakeholders
bullSoftware is created by people People set the contextbullTesting finds bugs acting as a skilled mental activitybullKey Question What testing would be most valuable right nowbullExpect changes Adapt testing plans based on test resultsbullTesting research requires empirical and psychological study
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 14110
13 General testing principles
14
bull Testing shows presence of defects but cannot prove that there are no moredefects testing can only reduce the probability of undiscovered defects
bull Complete exhaustive testing is impossible good strategy and risk managementmust be used
bull Pareto rule (defect clustering) usually 20 of the modules contain 80 of thebugs
bull Early testing testing activities should start as soon as possible (including hereplanning design reviews)
bull Pesticide paradox if the same set of tests are repeated over again no new bugswill be found the test cases should be reviewed modified and new test casesdeveloped
bull Context dependence test design and execution is context dependent (desktopweb applications real-time hellip)
bull Verification and Validation discovering defects cannot help a product that is not fitto the users needs
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 15110
13 General testing principles ndash heuristics of software testing
bullOperability - The better it works the more efficiently it can be tested
bullObservability - What we see is what we test
bull Controllability - The better we control the software the more the testing processcan be automated and optimized
bull Decomposability - By controlling the scope of testing we can quickly isolateproblems and perform effective and efficient testing
bull Simplicity - The less there is to test the more quickly we can test it
bull Stability - The fewer the changes the fewer are the disruptions to testing
bull Understandability - The more information we will have the smarter we will test
bull Suitability - The more we know about the intended use of the software the
better we can organize our testing to find important bugs15
1 4 1 F d t l t t h
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 16110
141 Fundamental test process ndash phases
-Test Planning amp Test control
-Test Analysis amp Design
-Test Implementation ampExecution
-Evaluating exit criteria ampReporting
-Test Closure activities
16
1 4 1 Fundamental test process planning amp control
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 17110
141 Fundamental test process ndash planning amp control
Planning1 Determine scope
bull Study project documents used software life-cycle specifications product desiredquality attributesbull Clarify test process expectations
2 Determine risksbull Choose quality risk analysis method (eg FMEA)
bull Document the list of risks probability impact priority identify mitigation actions3 Estimate testing effort determine costs develop schedule
bull Define necessary rolesbull Decompose test project into phases and tasks (WBS)bull Schedule tasks assign resources set-up dependencies
4 Refine planbull Select test strategy (how to do it what test types at which test levels)bull Select metrics to be used for defect tracking coverage monitoringbull Define entry and exit criteria
Controlbull Measure and analyze resultsbull Monitor testing progress coverage exit criteriabull Assign or reallocate resources update the test plan schedulebull Initiate corrective actionsbull Make decisions
17
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 18110
142 Fundamental test process ndash analysis amp design
bull Reviewing the test basis (such as requirements architecture designinterfaces)
bull Identifying test conditions or test requirements and required test data
based on analysis of test items and the specificationbull Designing the testsbull Choose test techniquesbull Identify test scenarios pre-conditions expected results post-
conditionsbull Identify possible test Oracles
bull Evaluating testability of the requirements and systembull Designing the test environment set-up and identifying any required
infrastructure and tools
(see Lee Copeland (b2))
18
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 19110
142 Fundamental test process ndash what is a test Oracle
19
bull The expected result (test outcome) must be defined at the test analysis stagebull Who will decide that (expected result = actual result) when the test will be
executed The Test Oracle
Test Oracle = A source to determine expected result a principle or mechanism torecognize a problem The Test Oracle can bebull an existing system (the old versionhellip)bull a document (specification user manual)bull a competent client representative
hellip but never the source code itself Oracles in use = Simplification of Risk do not assess lsquopass - failrsquo but instead
lsquoproblem - no problemrsquo
Problem Oracles and Automation - Our ability to automate testing isfundamentally constrained by our ability to create and use oracles
Possible issues
bull false alarmsbull missed bugs
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 20110
143 Fundamental test process ndash implementation amp execution
Test implementation
bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)
bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission
(see Rex Black (b4))
20
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 21110
143 Fundamental test process ndash priori tizing the Test Cases
Why prioritize the Test Cases
bullIt is not possible to test everything we must do our best in the time available
bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence
bullThis means we must prioritise and focus testing on the priorities
What to watch
bullSeverity of possible defectsbullProbability of possible defects
bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity
21
144 Fundamental test process ndash evaluating exit criteria and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 22110
144 Fundamental test process evaluating exit criteria andreporting
Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed
bull Check if testing mission should be changed
Test report ing
bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity
(see Rex Black (b4) amp RUP- Test discipline(s5))
22
1 4 5 F d l l
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 23110
145 Fundamental test process ndash test closure
bull Verify if test deliverables have been delivered
bull Check and close the remaining active bug reports
bull Archiving the test-ware and environment
bull Handover of the test environment
bull Analyze the identified test process problems (lessons learned)
bull Implement action plan based improvements
(see Rex Black (b4))
23
1 5 Th h l f i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 24110
15 The psychology of testing
24
Testing is regarded as a lsquodestructiversquo activity
(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts
bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise
bullShould be a planned organised kind ofperson
Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices
Rex BlackrsquosTop 10 professional errors
bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations
(see also Brian Marickrsquos article )
1 5 Th h l g f t ti g
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 25110
15 The psychology of testing
25
ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)
ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects
bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs
Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)
Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts
bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence
2 1 1 The V testing model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 26110
211 The V testing model
26
211 The V testing model ndash Verification amp Validationifi i C fi i b
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 27110
27
Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled
Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled
Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level
2 1 1 The W testing model ndash dynamic testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 28110
211 The W testing model ndash dynamic testing
28
2 1 1 The W testing model ndash static testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 29110
211 The W testing model static testing
29
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 30110
212 Software development models Waterfall
30
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 31110
212 Software development models Waterfall
Waterfall weaknesses
middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule
middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover
middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen
middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to
partition the system for delivery of pieces of the system
31
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 32110
p p yp
32
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 33110
p p yp
Rapid Prototype Model weaknesses
middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked
middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products
middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations
middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary
middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study
33
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 34110
34
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 35110
35
Incremental Model weaknesses
middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments
middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-
defined interfaces are required
middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 36110
36
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 37110
Spiral Model weaknesses
middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to
proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk
analysis and prototyping may be excessive
37
212 Software development models - Rational Unified Process
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 38110
38
213 Software development models ndash Testing life cycle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 39110
bull For each software activity there must be a corresponding testing activity
bull The objectives of the testing are specific to that ldquotestedrdquo activity
bull Plan analysis and design of a testing activity should be done during thecorresponding development activity
bull Review and inspection must be considered as part of testing activities
39
221 Test levels ndash Component testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 13110
12 What is testing ndash Testing ldquo schoolsrdquo
13
Analytic School - testing is rigorous academic and technicalbullTesting is a branch of CSMathematicsbullTesting techniques must have a logic-mathematical formbullKey Question Which techniques should we usebullRequire precise and detailed specifications
Factory School - testing is a way to measure progress with emphasis on cost and
repeatable standardsbullTesting must be managed amp cost effectivebullTesting validates the product amp measures development progressbullKey Questions How can we measure whether wersquore making progress When will we be donebullRequire clear boundaries between testing and other activities (startstop criteria)
bullEncourage standards (V-model) ldquobest practicesrdquo and certificationQuality School - emphasizes process amp quality acting as the gatekeeper bullSoftware quality requires disciplinebullTesters may need to police developers to follow the rulesbullKey Question Are we following a good process
bullTesting is a stepping stone to ldquoprocess improvementrdquoContext-Driven School - emphasizes people setting out to find the bugs that will bemost important to stakeholders
bullSoftware is created by people People set the contextbullTesting finds bugs acting as a skilled mental activitybullKey Question What testing would be most valuable right nowbullExpect changes Adapt testing plans based on test resultsbullTesting research requires empirical and psychological study
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 14110
13 General testing principles
14
bull Testing shows presence of defects but cannot prove that there are no moredefects testing can only reduce the probability of undiscovered defects
bull Complete exhaustive testing is impossible good strategy and risk managementmust be used
bull Pareto rule (defect clustering) usually 20 of the modules contain 80 of thebugs
bull Early testing testing activities should start as soon as possible (including hereplanning design reviews)
bull Pesticide paradox if the same set of tests are repeated over again no new bugswill be found the test cases should be reviewed modified and new test casesdeveloped
bull Context dependence test design and execution is context dependent (desktopweb applications real-time hellip)
bull Verification and Validation discovering defects cannot help a product that is not fitto the users needs
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 15110
13 General testing principles ndash heuristics of software testing
bullOperability - The better it works the more efficiently it can be tested
bullObservability - What we see is what we test
bull Controllability - The better we control the software the more the testing processcan be automated and optimized
bull Decomposability - By controlling the scope of testing we can quickly isolateproblems and perform effective and efficient testing
bull Simplicity - The less there is to test the more quickly we can test it
bull Stability - The fewer the changes the fewer are the disruptions to testing
bull Understandability - The more information we will have the smarter we will test
bull Suitability - The more we know about the intended use of the software the
better we can organize our testing to find important bugs15
1 4 1 F d t l t t h
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 16110
141 Fundamental test process ndash phases
-Test Planning amp Test control
-Test Analysis amp Design
-Test Implementation ampExecution
-Evaluating exit criteria ampReporting
-Test Closure activities
16
1 4 1 Fundamental test process planning amp control
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 17110
141 Fundamental test process ndash planning amp control
Planning1 Determine scope
bull Study project documents used software life-cycle specifications product desiredquality attributesbull Clarify test process expectations
2 Determine risksbull Choose quality risk analysis method (eg FMEA)
bull Document the list of risks probability impact priority identify mitigation actions3 Estimate testing effort determine costs develop schedule
bull Define necessary rolesbull Decompose test project into phases and tasks (WBS)bull Schedule tasks assign resources set-up dependencies
4 Refine planbull Select test strategy (how to do it what test types at which test levels)bull Select metrics to be used for defect tracking coverage monitoringbull Define entry and exit criteria
Controlbull Measure and analyze resultsbull Monitor testing progress coverage exit criteriabull Assign or reallocate resources update the test plan schedulebull Initiate corrective actionsbull Make decisions
17
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 18110
142 Fundamental test process ndash analysis amp design
bull Reviewing the test basis (such as requirements architecture designinterfaces)
bull Identifying test conditions or test requirements and required test data
based on analysis of test items and the specificationbull Designing the testsbull Choose test techniquesbull Identify test scenarios pre-conditions expected results post-
conditionsbull Identify possible test Oracles
bull Evaluating testability of the requirements and systembull Designing the test environment set-up and identifying any required
infrastructure and tools
(see Lee Copeland (b2))
18
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 19110
142 Fundamental test process ndash what is a test Oracle
19
bull The expected result (test outcome) must be defined at the test analysis stagebull Who will decide that (expected result = actual result) when the test will be
executed The Test Oracle
Test Oracle = A source to determine expected result a principle or mechanism torecognize a problem The Test Oracle can bebull an existing system (the old versionhellip)bull a document (specification user manual)bull a competent client representative
hellip but never the source code itself Oracles in use = Simplification of Risk do not assess lsquopass - failrsquo but instead
lsquoproblem - no problemrsquo
Problem Oracles and Automation - Our ability to automate testing isfundamentally constrained by our ability to create and use oracles
Possible issues
bull false alarmsbull missed bugs
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 20110
143 Fundamental test process ndash implementation amp execution
Test implementation
bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)
bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission
(see Rex Black (b4))
20
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 21110
143 Fundamental test process ndash priori tizing the Test Cases
Why prioritize the Test Cases
bullIt is not possible to test everything we must do our best in the time available
bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence
bullThis means we must prioritise and focus testing on the priorities
What to watch
bullSeverity of possible defectsbullProbability of possible defects
bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity
21
144 Fundamental test process ndash evaluating exit criteria and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 22110
144 Fundamental test process evaluating exit criteria andreporting
Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed
bull Check if testing mission should be changed
Test report ing
bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity
(see Rex Black (b4) amp RUP- Test discipline(s5))
22
1 4 5 F d l l
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 23110
145 Fundamental test process ndash test closure
bull Verify if test deliverables have been delivered
bull Check and close the remaining active bug reports
bull Archiving the test-ware and environment
bull Handover of the test environment
bull Analyze the identified test process problems (lessons learned)
bull Implement action plan based improvements
(see Rex Black (b4))
23
1 5 Th h l f i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 24110
15 The psychology of testing
24
Testing is regarded as a lsquodestructiversquo activity
(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts
bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise
bullShould be a planned organised kind ofperson
Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices
Rex BlackrsquosTop 10 professional errors
bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations
(see also Brian Marickrsquos article )
1 5 Th h l g f t ti g
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 25110
15 The psychology of testing
25
ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)
ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects
bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs
Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)
Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts
bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence
2 1 1 The V testing model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 26110
211 The V testing model
26
211 The V testing model ndash Verification amp Validationifi i C fi i b
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 27110
27
Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled
Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled
Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level
2 1 1 The W testing model ndash dynamic testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 28110
211 The W testing model ndash dynamic testing
28
2 1 1 The W testing model ndash static testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 29110
211 The W testing model static testing
29
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 30110
212 Software development models Waterfall
30
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 31110
212 Software development models Waterfall
Waterfall weaknesses
middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule
middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover
middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen
middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to
partition the system for delivery of pieces of the system
31
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 32110
p p yp
32
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 33110
p p yp
Rapid Prototype Model weaknesses
middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked
middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products
middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations
middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary
middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study
33
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 34110
34
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 35110
35
Incremental Model weaknesses
middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments
middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-
defined interfaces are required
middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 36110
36
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 37110
Spiral Model weaknesses
middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to
proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk
analysis and prototyping may be excessive
37
212 Software development models - Rational Unified Process
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 38110
38
213 Software development models ndash Testing life cycle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 39110
bull For each software activity there must be a corresponding testing activity
bull The objectives of the testing are specific to that ldquotestedrdquo activity
bull Plan analysis and design of a testing activity should be done during thecorresponding development activity
bull Review and inspection must be considered as part of testing activities
39
221 Test levels ndash Component testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 14110
13 General testing principles
14
bull Testing shows presence of defects but cannot prove that there are no moredefects testing can only reduce the probability of undiscovered defects
bull Complete exhaustive testing is impossible good strategy and risk managementmust be used
bull Pareto rule (defect clustering) usually 20 of the modules contain 80 of thebugs
bull Early testing testing activities should start as soon as possible (including hereplanning design reviews)
bull Pesticide paradox if the same set of tests are repeated over again no new bugswill be found the test cases should be reviewed modified and new test casesdeveloped
bull Context dependence test design and execution is context dependent (desktopweb applications real-time hellip)
bull Verification and Validation discovering defects cannot help a product that is not fitto the users needs
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 15110
13 General testing principles ndash heuristics of software testing
bullOperability - The better it works the more efficiently it can be tested
bullObservability - What we see is what we test
bull Controllability - The better we control the software the more the testing processcan be automated and optimized
bull Decomposability - By controlling the scope of testing we can quickly isolateproblems and perform effective and efficient testing
bull Simplicity - The less there is to test the more quickly we can test it
bull Stability - The fewer the changes the fewer are the disruptions to testing
bull Understandability - The more information we will have the smarter we will test
bull Suitability - The more we know about the intended use of the software the
better we can organize our testing to find important bugs15
1 4 1 F d t l t t h
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 16110
141 Fundamental test process ndash phases
-Test Planning amp Test control
-Test Analysis amp Design
-Test Implementation ampExecution
-Evaluating exit criteria ampReporting
-Test Closure activities
16
1 4 1 Fundamental test process planning amp control
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 17110
141 Fundamental test process ndash planning amp control
Planning1 Determine scope
bull Study project documents used software life-cycle specifications product desiredquality attributesbull Clarify test process expectations
2 Determine risksbull Choose quality risk analysis method (eg FMEA)
bull Document the list of risks probability impact priority identify mitigation actions3 Estimate testing effort determine costs develop schedule
bull Define necessary rolesbull Decompose test project into phases and tasks (WBS)bull Schedule tasks assign resources set-up dependencies
4 Refine planbull Select test strategy (how to do it what test types at which test levels)bull Select metrics to be used for defect tracking coverage monitoringbull Define entry and exit criteria
Controlbull Measure and analyze resultsbull Monitor testing progress coverage exit criteriabull Assign or reallocate resources update the test plan schedulebull Initiate corrective actionsbull Make decisions
17
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 18110
142 Fundamental test process ndash analysis amp design
bull Reviewing the test basis (such as requirements architecture designinterfaces)
bull Identifying test conditions or test requirements and required test data
based on analysis of test items and the specificationbull Designing the testsbull Choose test techniquesbull Identify test scenarios pre-conditions expected results post-
conditionsbull Identify possible test Oracles
bull Evaluating testability of the requirements and systembull Designing the test environment set-up and identifying any required
infrastructure and tools
(see Lee Copeland (b2))
18
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 19110
142 Fundamental test process ndash what is a test Oracle
19
bull The expected result (test outcome) must be defined at the test analysis stagebull Who will decide that (expected result = actual result) when the test will be
executed The Test Oracle
Test Oracle = A source to determine expected result a principle or mechanism torecognize a problem The Test Oracle can bebull an existing system (the old versionhellip)bull a document (specification user manual)bull a competent client representative
hellip but never the source code itself Oracles in use = Simplification of Risk do not assess lsquopass - failrsquo but instead
lsquoproblem - no problemrsquo
Problem Oracles and Automation - Our ability to automate testing isfundamentally constrained by our ability to create and use oracles
Possible issues
bull false alarmsbull missed bugs
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 20110
143 Fundamental test process ndash implementation amp execution
Test implementation
bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)
bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission
(see Rex Black (b4))
20
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 21110
143 Fundamental test process ndash priori tizing the Test Cases
Why prioritize the Test Cases
bullIt is not possible to test everything we must do our best in the time available
bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence
bullThis means we must prioritise and focus testing on the priorities
What to watch
bullSeverity of possible defectsbullProbability of possible defects
bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity
21
144 Fundamental test process ndash evaluating exit criteria and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 22110
144 Fundamental test process evaluating exit criteria andreporting
Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed
bull Check if testing mission should be changed
Test report ing
bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity
(see Rex Black (b4) amp RUP- Test discipline(s5))
22
1 4 5 F d l l
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 23110
145 Fundamental test process ndash test closure
bull Verify if test deliverables have been delivered
bull Check and close the remaining active bug reports
bull Archiving the test-ware and environment
bull Handover of the test environment
bull Analyze the identified test process problems (lessons learned)
bull Implement action plan based improvements
(see Rex Black (b4))
23
1 5 Th h l f i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 24110
15 The psychology of testing
24
Testing is regarded as a lsquodestructiversquo activity
(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts
bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise
bullShould be a planned organised kind ofperson
Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices
Rex BlackrsquosTop 10 professional errors
bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations
(see also Brian Marickrsquos article )
1 5 Th h l g f t ti g
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 25110
15 The psychology of testing
25
ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)
ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects
bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs
Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)
Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts
bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence
2 1 1 The V testing model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 26110
211 The V testing model
26
211 The V testing model ndash Verification amp Validationifi i C fi i b
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 27110
27
Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled
Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled
Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level
2 1 1 The W testing model ndash dynamic testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 28110
211 The W testing model ndash dynamic testing
28
2 1 1 The W testing model ndash static testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 29110
211 The W testing model static testing
29
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 30110
212 Software development models Waterfall
30
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 31110
212 Software development models Waterfall
Waterfall weaknesses
middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule
middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover
middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen
middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to
partition the system for delivery of pieces of the system
31
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 32110
p p yp
32
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 33110
p p yp
Rapid Prototype Model weaknesses
middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked
middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products
middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations
middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary
middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study
33
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 34110
34
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 35110
35
Incremental Model weaknesses
middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments
middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-
defined interfaces are required
middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 36110
36
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 37110
Spiral Model weaknesses
middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to
proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk
analysis and prototyping may be excessive
37
212 Software development models - Rational Unified Process
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 38110
38
213 Software development models ndash Testing life cycle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 39110
bull For each software activity there must be a corresponding testing activity
bull The objectives of the testing are specific to that ldquotestedrdquo activity
bull Plan analysis and design of a testing activity should be done during thecorresponding development activity
bull Review and inspection must be considered as part of testing activities
39
221 Test levels ndash Component testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 15110
13 General testing principles ndash heuristics of software testing
bullOperability - The better it works the more efficiently it can be tested
bullObservability - What we see is what we test
bull Controllability - The better we control the software the more the testing processcan be automated and optimized
bull Decomposability - By controlling the scope of testing we can quickly isolateproblems and perform effective and efficient testing
bull Simplicity - The less there is to test the more quickly we can test it
bull Stability - The fewer the changes the fewer are the disruptions to testing
bull Understandability - The more information we will have the smarter we will test
bull Suitability - The more we know about the intended use of the software the
better we can organize our testing to find important bugs15
1 4 1 F d t l t t h
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 16110
141 Fundamental test process ndash phases
-Test Planning amp Test control
-Test Analysis amp Design
-Test Implementation ampExecution
-Evaluating exit criteria ampReporting
-Test Closure activities
16
1 4 1 Fundamental test process planning amp control
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 17110
141 Fundamental test process ndash planning amp control
Planning1 Determine scope
bull Study project documents used software life-cycle specifications product desiredquality attributesbull Clarify test process expectations
2 Determine risksbull Choose quality risk analysis method (eg FMEA)
bull Document the list of risks probability impact priority identify mitigation actions3 Estimate testing effort determine costs develop schedule
bull Define necessary rolesbull Decompose test project into phases and tasks (WBS)bull Schedule tasks assign resources set-up dependencies
4 Refine planbull Select test strategy (how to do it what test types at which test levels)bull Select metrics to be used for defect tracking coverage monitoringbull Define entry and exit criteria
Controlbull Measure and analyze resultsbull Monitor testing progress coverage exit criteriabull Assign or reallocate resources update the test plan schedulebull Initiate corrective actionsbull Make decisions
17
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 18110
142 Fundamental test process ndash analysis amp design
bull Reviewing the test basis (such as requirements architecture designinterfaces)
bull Identifying test conditions or test requirements and required test data
based on analysis of test items and the specificationbull Designing the testsbull Choose test techniquesbull Identify test scenarios pre-conditions expected results post-
conditionsbull Identify possible test Oracles
bull Evaluating testability of the requirements and systembull Designing the test environment set-up and identifying any required
infrastructure and tools
(see Lee Copeland (b2))
18
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 19110
142 Fundamental test process ndash what is a test Oracle
19
bull The expected result (test outcome) must be defined at the test analysis stagebull Who will decide that (expected result = actual result) when the test will be
executed The Test Oracle
Test Oracle = A source to determine expected result a principle or mechanism torecognize a problem The Test Oracle can bebull an existing system (the old versionhellip)bull a document (specification user manual)bull a competent client representative
hellip but never the source code itself Oracles in use = Simplification of Risk do not assess lsquopass - failrsquo but instead
lsquoproblem - no problemrsquo
Problem Oracles and Automation - Our ability to automate testing isfundamentally constrained by our ability to create and use oracles
Possible issues
bull false alarmsbull missed bugs
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 20110
143 Fundamental test process ndash implementation amp execution
Test implementation
bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)
bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission
(see Rex Black (b4))
20
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 21110
143 Fundamental test process ndash priori tizing the Test Cases
Why prioritize the Test Cases
bullIt is not possible to test everything we must do our best in the time available
bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence
bullThis means we must prioritise and focus testing on the priorities
What to watch
bullSeverity of possible defectsbullProbability of possible defects
bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity
21
144 Fundamental test process ndash evaluating exit criteria and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 22110
144 Fundamental test process evaluating exit criteria andreporting
Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed
bull Check if testing mission should be changed
Test report ing
bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity
(see Rex Black (b4) amp RUP- Test discipline(s5))
22
1 4 5 F d l l
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 23110
145 Fundamental test process ndash test closure
bull Verify if test deliverables have been delivered
bull Check and close the remaining active bug reports
bull Archiving the test-ware and environment
bull Handover of the test environment
bull Analyze the identified test process problems (lessons learned)
bull Implement action plan based improvements
(see Rex Black (b4))
23
1 5 Th h l f i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 24110
15 The psychology of testing
24
Testing is regarded as a lsquodestructiversquo activity
(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts
bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise
bullShould be a planned organised kind ofperson
Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices
Rex BlackrsquosTop 10 professional errors
bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations
(see also Brian Marickrsquos article )
1 5 Th h l g f t ti g
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 25110
15 The psychology of testing
25
ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)
ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects
bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs
Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)
Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts
bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence
2 1 1 The V testing model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 26110
211 The V testing model
26
211 The V testing model ndash Verification amp Validationifi i C fi i b
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 27110
27
Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled
Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled
Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level
2 1 1 The W testing model ndash dynamic testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 28110
211 The W testing model ndash dynamic testing
28
2 1 1 The W testing model ndash static testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 29110
211 The W testing model static testing
29
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 30110
212 Software development models Waterfall
30
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 31110
212 Software development models Waterfall
Waterfall weaknesses
middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule
middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover
middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen
middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to
partition the system for delivery of pieces of the system
31
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 32110
p p yp
32
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 33110
p p yp
Rapid Prototype Model weaknesses
middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked
middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products
middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations
middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary
middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study
33
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 34110
34
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 35110
35
Incremental Model weaknesses
middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments
middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-
defined interfaces are required
middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 36110
36
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 37110
Spiral Model weaknesses
middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to
proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk
analysis and prototyping may be excessive
37
212 Software development models - Rational Unified Process
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 38110
38
213 Software development models ndash Testing life cycle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 39110
bull For each software activity there must be a corresponding testing activity
bull The objectives of the testing are specific to that ldquotestedrdquo activity
bull Plan analysis and design of a testing activity should be done during thecorresponding development activity
bull Review and inspection must be considered as part of testing activities
39
221 Test levels ndash Component testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 16110
141 Fundamental test process ndash phases
-Test Planning amp Test control
-Test Analysis amp Design
-Test Implementation ampExecution
-Evaluating exit criteria ampReporting
-Test Closure activities
16
1 4 1 Fundamental test process planning amp control
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 17110
141 Fundamental test process ndash planning amp control
Planning1 Determine scope
bull Study project documents used software life-cycle specifications product desiredquality attributesbull Clarify test process expectations
2 Determine risksbull Choose quality risk analysis method (eg FMEA)
bull Document the list of risks probability impact priority identify mitigation actions3 Estimate testing effort determine costs develop schedule
bull Define necessary rolesbull Decompose test project into phases and tasks (WBS)bull Schedule tasks assign resources set-up dependencies
4 Refine planbull Select test strategy (how to do it what test types at which test levels)bull Select metrics to be used for defect tracking coverage monitoringbull Define entry and exit criteria
Controlbull Measure and analyze resultsbull Monitor testing progress coverage exit criteriabull Assign or reallocate resources update the test plan schedulebull Initiate corrective actionsbull Make decisions
17
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 18110
142 Fundamental test process ndash analysis amp design
bull Reviewing the test basis (such as requirements architecture designinterfaces)
bull Identifying test conditions or test requirements and required test data
based on analysis of test items and the specificationbull Designing the testsbull Choose test techniquesbull Identify test scenarios pre-conditions expected results post-
conditionsbull Identify possible test Oracles
bull Evaluating testability of the requirements and systembull Designing the test environment set-up and identifying any required
infrastructure and tools
(see Lee Copeland (b2))
18
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 19110
142 Fundamental test process ndash what is a test Oracle
19
bull The expected result (test outcome) must be defined at the test analysis stagebull Who will decide that (expected result = actual result) when the test will be
executed The Test Oracle
Test Oracle = A source to determine expected result a principle or mechanism torecognize a problem The Test Oracle can bebull an existing system (the old versionhellip)bull a document (specification user manual)bull a competent client representative
hellip but never the source code itself Oracles in use = Simplification of Risk do not assess lsquopass - failrsquo but instead
lsquoproblem - no problemrsquo
Problem Oracles and Automation - Our ability to automate testing isfundamentally constrained by our ability to create and use oracles
Possible issues
bull false alarmsbull missed bugs
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 20110
143 Fundamental test process ndash implementation amp execution
Test implementation
bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)
bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission
(see Rex Black (b4))
20
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 21110
143 Fundamental test process ndash priori tizing the Test Cases
Why prioritize the Test Cases
bullIt is not possible to test everything we must do our best in the time available
bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence
bullThis means we must prioritise and focus testing on the priorities
What to watch
bullSeverity of possible defectsbullProbability of possible defects
bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity
21
144 Fundamental test process ndash evaluating exit criteria and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 22110
144 Fundamental test process evaluating exit criteria andreporting
Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed
bull Check if testing mission should be changed
Test report ing
bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity
(see Rex Black (b4) amp RUP- Test discipline(s5))
22
1 4 5 F d l l
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 23110
145 Fundamental test process ndash test closure
bull Verify if test deliverables have been delivered
bull Check and close the remaining active bug reports
bull Archiving the test-ware and environment
bull Handover of the test environment
bull Analyze the identified test process problems (lessons learned)
bull Implement action plan based improvements
(see Rex Black (b4))
23
1 5 Th h l f i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 24110
15 The psychology of testing
24
Testing is regarded as a lsquodestructiversquo activity
(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts
bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise
bullShould be a planned organised kind ofperson
Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices
Rex BlackrsquosTop 10 professional errors
bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations
(see also Brian Marickrsquos article )
1 5 Th h l g f t ti g
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 25110
15 The psychology of testing
25
ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)
ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects
bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs
Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)
Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts
bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence
2 1 1 The V testing model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 26110
211 The V testing model
26
211 The V testing model ndash Verification amp Validationifi i C fi i b
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 27110
27
Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled
Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled
Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level
2 1 1 The W testing model ndash dynamic testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 28110
211 The W testing model ndash dynamic testing
28
2 1 1 The W testing model ndash static testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 29110
211 The W testing model static testing
29
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 30110
212 Software development models Waterfall
30
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 31110
212 Software development models Waterfall
Waterfall weaknesses
middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule
middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover
middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen
middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to
partition the system for delivery of pieces of the system
31
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 32110
p p yp
32
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 33110
p p yp
Rapid Prototype Model weaknesses
middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked
middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products
middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations
middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary
middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study
33
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 34110
34
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 35110
35
Incremental Model weaknesses
middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments
middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-
defined interfaces are required
middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 36110
36
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 37110
Spiral Model weaknesses
middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to
proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk
analysis and prototyping may be excessive
37
212 Software development models - Rational Unified Process
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 38110
38
213 Software development models ndash Testing life cycle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 39110
bull For each software activity there must be a corresponding testing activity
bull The objectives of the testing are specific to that ldquotestedrdquo activity
bull Plan analysis and design of a testing activity should be done during thecorresponding development activity
bull Review and inspection must be considered as part of testing activities
39
221 Test levels ndash Component testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 17110
141 Fundamental test process ndash planning amp control
Planning1 Determine scope
bull Study project documents used software life-cycle specifications product desiredquality attributesbull Clarify test process expectations
2 Determine risksbull Choose quality risk analysis method (eg FMEA)
bull Document the list of risks probability impact priority identify mitigation actions3 Estimate testing effort determine costs develop schedule
bull Define necessary rolesbull Decompose test project into phases and tasks (WBS)bull Schedule tasks assign resources set-up dependencies
4 Refine planbull Select test strategy (how to do it what test types at which test levels)bull Select metrics to be used for defect tracking coverage monitoringbull Define entry and exit criteria
Controlbull Measure and analyze resultsbull Monitor testing progress coverage exit criteriabull Assign or reallocate resources update the test plan schedulebull Initiate corrective actionsbull Make decisions
17
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 18110
142 Fundamental test process ndash analysis amp design
bull Reviewing the test basis (such as requirements architecture designinterfaces)
bull Identifying test conditions or test requirements and required test data
based on analysis of test items and the specificationbull Designing the testsbull Choose test techniquesbull Identify test scenarios pre-conditions expected results post-
conditionsbull Identify possible test Oracles
bull Evaluating testability of the requirements and systembull Designing the test environment set-up and identifying any required
infrastructure and tools
(see Lee Copeland (b2))
18
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 19110
142 Fundamental test process ndash what is a test Oracle
19
bull The expected result (test outcome) must be defined at the test analysis stagebull Who will decide that (expected result = actual result) when the test will be
executed The Test Oracle
Test Oracle = A source to determine expected result a principle or mechanism torecognize a problem The Test Oracle can bebull an existing system (the old versionhellip)bull a document (specification user manual)bull a competent client representative
hellip but never the source code itself Oracles in use = Simplification of Risk do not assess lsquopass - failrsquo but instead
lsquoproblem - no problemrsquo
Problem Oracles and Automation - Our ability to automate testing isfundamentally constrained by our ability to create and use oracles
Possible issues
bull false alarmsbull missed bugs
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 20110
143 Fundamental test process ndash implementation amp execution
Test implementation
bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)
bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission
(see Rex Black (b4))
20
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 21110
143 Fundamental test process ndash priori tizing the Test Cases
Why prioritize the Test Cases
bullIt is not possible to test everything we must do our best in the time available
bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence
bullThis means we must prioritise and focus testing on the priorities
What to watch
bullSeverity of possible defectsbullProbability of possible defects
bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity
21
144 Fundamental test process ndash evaluating exit criteria and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 22110
144 Fundamental test process evaluating exit criteria andreporting
Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed
bull Check if testing mission should be changed
Test report ing
bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity
(see Rex Black (b4) amp RUP- Test discipline(s5))
22
1 4 5 F d l l
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 23110
145 Fundamental test process ndash test closure
bull Verify if test deliverables have been delivered
bull Check and close the remaining active bug reports
bull Archiving the test-ware and environment
bull Handover of the test environment
bull Analyze the identified test process problems (lessons learned)
bull Implement action plan based improvements
(see Rex Black (b4))
23
1 5 Th h l f i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 24110
15 The psychology of testing
24
Testing is regarded as a lsquodestructiversquo activity
(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts
bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise
bullShould be a planned organised kind ofperson
Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices
Rex BlackrsquosTop 10 professional errors
bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations
(see also Brian Marickrsquos article )
1 5 Th h l g f t ti g
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 25110
15 The psychology of testing
25
ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)
ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects
bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs
Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)
Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts
bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence
2 1 1 The V testing model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 26110
211 The V testing model
26
211 The V testing model ndash Verification amp Validationifi i C fi i b
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 27110
27
Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled
Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled
Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level
2 1 1 The W testing model ndash dynamic testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 28110
211 The W testing model ndash dynamic testing
28
2 1 1 The W testing model ndash static testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 29110
211 The W testing model static testing
29
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 30110
212 Software development models Waterfall
30
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 31110
212 Software development models Waterfall
Waterfall weaknesses
middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule
middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover
middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen
middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to
partition the system for delivery of pieces of the system
31
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 32110
p p yp
32
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 33110
p p yp
Rapid Prototype Model weaknesses
middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked
middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products
middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations
middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary
middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study
33
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 34110
34
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 35110
35
Incremental Model weaknesses
middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments
middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-
defined interfaces are required
middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 36110
36
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 37110
Spiral Model weaknesses
middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to
proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk
analysis and prototyping may be excessive
37
212 Software development models - Rational Unified Process
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 38110
38
213 Software development models ndash Testing life cycle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 39110
bull For each software activity there must be a corresponding testing activity
bull The objectives of the testing are specific to that ldquotestedrdquo activity
bull Plan analysis and design of a testing activity should be done during thecorresponding development activity
bull Review and inspection must be considered as part of testing activities
39
221 Test levels ndash Component testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 18110
142 Fundamental test process ndash analysis amp design
bull Reviewing the test basis (such as requirements architecture designinterfaces)
bull Identifying test conditions or test requirements and required test data
based on analysis of test items and the specificationbull Designing the testsbull Choose test techniquesbull Identify test scenarios pre-conditions expected results post-
conditionsbull Identify possible test Oracles
bull Evaluating testability of the requirements and systembull Designing the test environment set-up and identifying any required
infrastructure and tools
(see Lee Copeland (b2))
18
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 19110
142 Fundamental test process ndash what is a test Oracle
19
bull The expected result (test outcome) must be defined at the test analysis stagebull Who will decide that (expected result = actual result) when the test will be
executed The Test Oracle
Test Oracle = A source to determine expected result a principle or mechanism torecognize a problem The Test Oracle can bebull an existing system (the old versionhellip)bull a document (specification user manual)bull a competent client representative
hellip but never the source code itself Oracles in use = Simplification of Risk do not assess lsquopass - failrsquo but instead
lsquoproblem - no problemrsquo
Problem Oracles and Automation - Our ability to automate testing isfundamentally constrained by our ability to create and use oracles
Possible issues
bull false alarmsbull missed bugs
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 20110
143 Fundamental test process ndash implementation amp execution
Test implementation
bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)
bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission
(see Rex Black (b4))
20
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 21110
143 Fundamental test process ndash priori tizing the Test Cases
Why prioritize the Test Cases
bullIt is not possible to test everything we must do our best in the time available
bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence
bullThis means we must prioritise and focus testing on the priorities
What to watch
bullSeverity of possible defectsbullProbability of possible defects
bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity
21
144 Fundamental test process ndash evaluating exit criteria and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 22110
144 Fundamental test process evaluating exit criteria andreporting
Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed
bull Check if testing mission should be changed
Test report ing
bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity
(see Rex Black (b4) amp RUP- Test discipline(s5))
22
1 4 5 F d l l
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 23110
145 Fundamental test process ndash test closure
bull Verify if test deliverables have been delivered
bull Check and close the remaining active bug reports
bull Archiving the test-ware and environment
bull Handover of the test environment
bull Analyze the identified test process problems (lessons learned)
bull Implement action plan based improvements
(see Rex Black (b4))
23
1 5 Th h l f i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 24110
15 The psychology of testing
24
Testing is regarded as a lsquodestructiversquo activity
(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts
bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise
bullShould be a planned organised kind ofperson
Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices
Rex BlackrsquosTop 10 professional errors
bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations
(see also Brian Marickrsquos article )
1 5 Th h l g f t ti g
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 25110
15 The psychology of testing
25
ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)
ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects
bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs
Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)
Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts
bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence
2 1 1 The V testing model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 26110
211 The V testing model
26
211 The V testing model ndash Verification amp Validationifi i C fi i b
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 27110
27
Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled
Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled
Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level
2 1 1 The W testing model ndash dynamic testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 28110
211 The W testing model ndash dynamic testing
28
2 1 1 The W testing model ndash static testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 29110
211 The W testing model static testing
29
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 30110
212 Software development models Waterfall
30
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 31110
212 Software development models Waterfall
Waterfall weaknesses
middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule
middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover
middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen
middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to
partition the system for delivery of pieces of the system
31
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 32110
p p yp
32
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 33110
p p yp
Rapid Prototype Model weaknesses
middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked
middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products
middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations
middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary
middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study
33
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 34110
34
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 35110
35
Incremental Model weaknesses
middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments
middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-
defined interfaces are required
middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 36110
36
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 37110
Spiral Model weaknesses
middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to
proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk
analysis and prototyping may be excessive
37
212 Software development models - Rational Unified Process
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 38110
38
213 Software development models ndash Testing life cycle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 39110
bull For each software activity there must be a corresponding testing activity
bull The objectives of the testing are specific to that ldquotestedrdquo activity
bull Plan analysis and design of a testing activity should be done during thecorresponding development activity
bull Review and inspection must be considered as part of testing activities
39
221 Test levels ndash Component testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 19110
142 Fundamental test process ndash what is a test Oracle
19
bull The expected result (test outcome) must be defined at the test analysis stagebull Who will decide that (expected result = actual result) when the test will be
executed The Test Oracle
Test Oracle = A source to determine expected result a principle or mechanism torecognize a problem The Test Oracle can bebull an existing system (the old versionhellip)bull a document (specification user manual)bull a competent client representative
hellip but never the source code itself Oracles in use = Simplification of Risk do not assess lsquopass - failrsquo but instead
lsquoproblem - no problemrsquo
Problem Oracles and Automation - Our ability to automate testing isfundamentally constrained by our ability to create and use oracles
Possible issues
bull false alarmsbull missed bugs
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 20110
143 Fundamental test process ndash implementation amp execution
Test implementation
bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)
bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission
(see Rex Black (b4))
20
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 21110
143 Fundamental test process ndash priori tizing the Test Cases
Why prioritize the Test Cases
bullIt is not possible to test everything we must do our best in the time available
bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence
bullThis means we must prioritise and focus testing on the priorities
What to watch
bullSeverity of possible defectsbullProbability of possible defects
bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity
21
144 Fundamental test process ndash evaluating exit criteria and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 22110
144 Fundamental test process evaluating exit criteria andreporting
Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed
bull Check if testing mission should be changed
Test report ing
bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity
(see Rex Black (b4) amp RUP- Test discipline(s5))
22
1 4 5 F d l l
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 23110
145 Fundamental test process ndash test closure
bull Verify if test deliverables have been delivered
bull Check and close the remaining active bug reports
bull Archiving the test-ware and environment
bull Handover of the test environment
bull Analyze the identified test process problems (lessons learned)
bull Implement action plan based improvements
(see Rex Black (b4))
23
1 5 Th h l f i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 24110
15 The psychology of testing
24
Testing is regarded as a lsquodestructiversquo activity
(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts
bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise
bullShould be a planned organised kind ofperson
Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices
Rex BlackrsquosTop 10 professional errors
bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations
(see also Brian Marickrsquos article )
1 5 Th h l g f t ti g
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 25110
15 The psychology of testing
25
ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)
ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects
bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs
Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)
Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts
bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence
2 1 1 The V testing model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 26110
211 The V testing model
26
211 The V testing model ndash Verification amp Validationifi i C fi i b
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 27110
27
Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled
Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled
Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level
2 1 1 The W testing model ndash dynamic testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 28110
211 The W testing model ndash dynamic testing
28
2 1 1 The W testing model ndash static testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 29110
211 The W testing model static testing
29
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 30110
212 Software development models Waterfall
30
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 31110
212 Software development models Waterfall
Waterfall weaknesses
middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule
middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover
middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen
middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to
partition the system for delivery of pieces of the system
31
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 32110
p p yp
32
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 33110
p p yp
Rapid Prototype Model weaknesses
middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked
middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products
middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations
middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary
middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study
33
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 34110
34
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 35110
35
Incremental Model weaknesses
middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments
middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-
defined interfaces are required
middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 36110
36
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 37110
Spiral Model weaknesses
middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to
proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk
analysis and prototyping may be excessive
37
212 Software development models - Rational Unified Process
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 38110
38
213 Software development models ndash Testing life cycle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 39110
bull For each software activity there must be a corresponding testing activity
bull The objectives of the testing are specific to that ldquotestedrdquo activity
bull Plan analysis and design of a testing activity should be done during thecorresponding development activity
bull Review and inspection must be considered as part of testing activities
39
221 Test levels ndash Component testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 20110
143 Fundamental test process ndash implementation amp execution
Test implementation
bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)
bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission
(see Rex Black (b4))
20
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 21110
143 Fundamental test process ndash priori tizing the Test Cases
Why prioritize the Test Cases
bullIt is not possible to test everything we must do our best in the time available
bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence
bullThis means we must prioritise and focus testing on the priorities
What to watch
bullSeverity of possible defectsbullProbability of possible defects
bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity
21
144 Fundamental test process ndash evaluating exit criteria and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 22110
144 Fundamental test process evaluating exit criteria andreporting
Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed
bull Check if testing mission should be changed
Test report ing
bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity
(see Rex Black (b4) amp RUP- Test discipline(s5))
22
1 4 5 F d l l
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 23110
145 Fundamental test process ndash test closure
bull Verify if test deliverables have been delivered
bull Check and close the remaining active bug reports
bull Archiving the test-ware and environment
bull Handover of the test environment
bull Analyze the identified test process problems (lessons learned)
bull Implement action plan based improvements
(see Rex Black (b4))
23
1 5 Th h l f i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 24110
15 The psychology of testing
24
Testing is regarded as a lsquodestructiversquo activity
(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts
bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise
bullShould be a planned organised kind ofperson
Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices
Rex BlackrsquosTop 10 professional errors
bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations
(see also Brian Marickrsquos article )
1 5 Th h l g f t ti g
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 25110
15 The psychology of testing
25
ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)
ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects
bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs
Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)
Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts
bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence
2 1 1 The V testing model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 26110
211 The V testing model
26
211 The V testing model ndash Verification amp Validationifi i C fi i b
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 27110
27
Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled
Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled
Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level
2 1 1 The W testing model ndash dynamic testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 28110
211 The W testing model ndash dynamic testing
28
2 1 1 The W testing model ndash static testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 29110
211 The W testing model static testing
29
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 30110
212 Software development models Waterfall
30
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 31110
212 Software development models Waterfall
Waterfall weaknesses
middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule
middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover
middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen
middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to
partition the system for delivery of pieces of the system
31
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 32110
p p yp
32
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 33110
p p yp
Rapid Prototype Model weaknesses
middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked
middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products
middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations
middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary
middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study
33
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 34110
34
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 35110
35
Incremental Model weaknesses
middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments
middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-
defined interfaces are required
middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 36110
36
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 37110
Spiral Model weaknesses
middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to
proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk
analysis and prototyping may be excessive
37
212 Software development models - Rational Unified Process
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 38110
38
213 Software development models ndash Testing life cycle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 39110
bull For each software activity there must be a corresponding testing activity
bull The objectives of the testing are specific to that ldquotestedrdquo activity
bull Plan analysis and design of a testing activity should be done during thecorresponding development activity
bull Review and inspection must be considered as part of testing activities
39
221 Test levels ndash Component testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 21110
143 Fundamental test process ndash priori tizing the Test Cases
Why prioritize the Test Cases
bullIt is not possible to test everything we must do our best in the time available
bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence
bullThis means we must prioritise and focus testing on the priorities
What to watch
bullSeverity of possible defectsbullProbability of possible defects
bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity
21
144 Fundamental test process ndash evaluating exit criteria and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 22110
144 Fundamental test process evaluating exit criteria andreporting
Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed
bull Check if testing mission should be changed
Test report ing
bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity
(see Rex Black (b4) amp RUP- Test discipline(s5))
22
1 4 5 F d l l
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 23110
145 Fundamental test process ndash test closure
bull Verify if test deliverables have been delivered
bull Check and close the remaining active bug reports
bull Archiving the test-ware and environment
bull Handover of the test environment
bull Analyze the identified test process problems (lessons learned)
bull Implement action plan based improvements
(see Rex Black (b4))
23
1 5 Th h l f i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 24110
15 The psychology of testing
24
Testing is regarded as a lsquodestructiversquo activity
(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts
bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise
bullShould be a planned organised kind ofperson
Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices
Rex BlackrsquosTop 10 professional errors
bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations
(see also Brian Marickrsquos article )
1 5 Th h l g f t ti g
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 25110
15 The psychology of testing
25
ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)
ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects
bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs
Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)
Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts
bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence
2 1 1 The V testing model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 26110
211 The V testing model
26
211 The V testing model ndash Verification amp Validationifi i C fi i b
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 27110
27
Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled
Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled
Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level
2 1 1 The W testing model ndash dynamic testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 28110
211 The W testing model ndash dynamic testing
28
2 1 1 The W testing model ndash static testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 29110
211 The W testing model static testing
29
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 30110
212 Software development models Waterfall
30
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 31110
212 Software development models Waterfall
Waterfall weaknesses
middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule
middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover
middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen
middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to
partition the system for delivery of pieces of the system
31
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 32110
p p yp
32
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 33110
p p yp
Rapid Prototype Model weaknesses
middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked
middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products
middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations
middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary
middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study
33
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 34110
34
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 35110
35
Incremental Model weaknesses
middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments
middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-
defined interfaces are required
middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 36110
36
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 37110
Spiral Model weaknesses
middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to
proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk
analysis and prototyping may be excessive
37
212 Software development models - Rational Unified Process
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 38110
38
213 Software development models ndash Testing life cycle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 39110
bull For each software activity there must be a corresponding testing activity
bull The objectives of the testing are specific to that ldquotestedrdquo activity
bull Plan analysis and design of a testing activity should be done during thecorresponding development activity
bull Review and inspection must be considered as part of testing activities
39
221 Test levels ndash Component testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 22110
144 Fundamental test process evaluating exit criteria andreporting
Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed
bull Check if testing mission should be changed
Test report ing
bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity
(see Rex Black (b4) amp RUP- Test discipline(s5))
22
1 4 5 F d l l
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 23110
145 Fundamental test process ndash test closure
bull Verify if test deliverables have been delivered
bull Check and close the remaining active bug reports
bull Archiving the test-ware and environment
bull Handover of the test environment
bull Analyze the identified test process problems (lessons learned)
bull Implement action plan based improvements
(see Rex Black (b4))
23
1 5 Th h l f i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 24110
15 The psychology of testing
24
Testing is regarded as a lsquodestructiversquo activity
(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts
bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise
bullShould be a planned organised kind ofperson
Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices
Rex BlackrsquosTop 10 professional errors
bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations
(see also Brian Marickrsquos article )
1 5 Th h l g f t ti g
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 25110
15 The psychology of testing
25
ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)
ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects
bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs
Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)
Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts
bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence
2 1 1 The V testing model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 26110
211 The V testing model
26
211 The V testing model ndash Verification amp Validationifi i C fi i b
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 27110
27
Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled
Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled
Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level
2 1 1 The W testing model ndash dynamic testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 28110
211 The W testing model ndash dynamic testing
28
2 1 1 The W testing model ndash static testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 29110
211 The W testing model static testing
29
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 30110
212 Software development models Waterfall
30
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 31110
212 Software development models Waterfall
Waterfall weaknesses
middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule
middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover
middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen
middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to
partition the system for delivery of pieces of the system
31
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 32110
p p yp
32
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 33110
p p yp
Rapid Prototype Model weaknesses
middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked
middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products
middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations
middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary
middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study
33
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 34110
34
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 35110
35
Incremental Model weaknesses
middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments
middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-
defined interfaces are required
middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 36110
36
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 37110
Spiral Model weaknesses
middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to
proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk
analysis and prototyping may be excessive
37
212 Software development models - Rational Unified Process
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 38110
38
213 Software development models ndash Testing life cycle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 39110
bull For each software activity there must be a corresponding testing activity
bull The objectives of the testing are specific to that ldquotestedrdquo activity
bull Plan analysis and design of a testing activity should be done during thecorresponding development activity
bull Review and inspection must be considered as part of testing activities
39
221 Test levels ndash Component testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 23110
145 Fundamental test process ndash test closure
bull Verify if test deliverables have been delivered
bull Check and close the remaining active bug reports
bull Archiving the test-ware and environment
bull Handover of the test environment
bull Analyze the identified test process problems (lessons learned)
bull Implement action plan based improvements
(see Rex Black (b4))
23
1 5 Th h l f i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 24110
15 The psychology of testing
24
Testing is regarded as a lsquodestructiversquo activity
(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts
bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise
bullShould be a planned organised kind ofperson
Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices
Rex BlackrsquosTop 10 professional errors
bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations
(see also Brian Marickrsquos article )
1 5 Th h l g f t ti g
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 25110
15 The psychology of testing
25
ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)
ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects
bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs
Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)
Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts
bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence
2 1 1 The V testing model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 26110
211 The V testing model
26
211 The V testing model ndash Verification amp Validationifi i C fi i b
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 27110
27
Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled
Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled
Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level
2 1 1 The W testing model ndash dynamic testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 28110
211 The W testing model ndash dynamic testing
28
2 1 1 The W testing model ndash static testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 29110
211 The W testing model static testing
29
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 30110
212 Software development models Waterfall
30
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 31110
212 Software development models Waterfall
Waterfall weaknesses
middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule
middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover
middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen
middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to
partition the system for delivery of pieces of the system
31
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 32110
p p yp
32
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 33110
p p yp
Rapid Prototype Model weaknesses
middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked
middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products
middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations
middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary
middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study
33
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 34110
34
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 35110
35
Incremental Model weaknesses
middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments
middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-
defined interfaces are required
middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 36110
36
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 37110
Spiral Model weaknesses
middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to
proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk
analysis and prototyping may be excessive
37
212 Software development models - Rational Unified Process
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 38110
38
213 Software development models ndash Testing life cycle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 39110
bull For each software activity there must be a corresponding testing activity
bull The objectives of the testing are specific to that ldquotestedrdquo activity
bull Plan analysis and design of a testing activity should be done during thecorresponding development activity
bull Review and inspection must be considered as part of testing activities
39
221 Test levels ndash Component testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 24110
15 The psychology of testing
24
Testing is regarded as a lsquodestructiversquo activity
(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts
bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise
bullShould be a planned organised kind ofperson
Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices
Rex BlackrsquosTop 10 professional errors
bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations
(see also Brian Marickrsquos article )
1 5 Th h l g f t ti g
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 25110
15 The psychology of testing
25
ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)
ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects
bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs
Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)
Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts
bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence
2 1 1 The V testing model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 26110
211 The V testing model
26
211 The V testing model ndash Verification amp Validationifi i C fi i b
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 27110
27
Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled
Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled
Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level
2 1 1 The W testing model ndash dynamic testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 28110
211 The W testing model ndash dynamic testing
28
2 1 1 The W testing model ndash static testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 29110
211 The W testing model static testing
29
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 30110
212 Software development models Waterfall
30
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 31110
212 Software development models Waterfall
Waterfall weaknesses
middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule
middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover
middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen
middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to
partition the system for delivery of pieces of the system
31
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 32110
p p yp
32
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 33110
p p yp
Rapid Prototype Model weaknesses
middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked
middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products
middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations
middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary
middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study
33
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 34110
34
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 35110
35
Incremental Model weaknesses
middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments
middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-
defined interfaces are required
middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 36110
36
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 37110
Spiral Model weaknesses
middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to
proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk
analysis and prototyping may be excessive
37
212 Software development models - Rational Unified Process
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 38110
38
213 Software development models ndash Testing life cycle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 39110
bull For each software activity there must be a corresponding testing activity
bull The objectives of the testing are specific to that ldquotestedrdquo activity
bull Plan analysis and design of a testing activity should be done during thecorresponding development activity
bull Review and inspection must be considered as part of testing activities
39
221 Test levels ndash Component testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 25110
15 The psychology of testing
25
ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)
ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects
bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs
Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)
Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts
bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence
2 1 1 The V testing model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 26110
211 The V testing model
26
211 The V testing model ndash Verification amp Validationifi i C fi i b
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 27110
27
Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled
Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled
Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level
2 1 1 The W testing model ndash dynamic testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 28110
211 The W testing model ndash dynamic testing
28
2 1 1 The W testing model ndash static testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 29110
211 The W testing model static testing
29
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 30110
212 Software development models Waterfall
30
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 31110
212 Software development models Waterfall
Waterfall weaknesses
middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule
middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover
middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen
middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to
partition the system for delivery of pieces of the system
31
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 32110
p p yp
32
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 33110
p p yp
Rapid Prototype Model weaknesses
middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked
middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products
middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations
middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary
middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study
33
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 34110
34
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 35110
35
Incremental Model weaknesses
middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments
middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-
defined interfaces are required
middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 36110
36
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 37110
Spiral Model weaknesses
middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to
proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk
analysis and prototyping may be excessive
37
212 Software development models - Rational Unified Process
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 38110
38
213 Software development models ndash Testing life cycle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 39110
bull For each software activity there must be a corresponding testing activity
bull The objectives of the testing are specific to that ldquotestedrdquo activity
bull Plan analysis and design of a testing activity should be done during thecorresponding development activity
bull Review and inspection must be considered as part of testing activities
39
221 Test levels ndash Component testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 26110
211 The V testing model
26
211 The V testing model ndash Verification amp Validationifi i C fi i b
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 27110
27
Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled
Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled
Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level
2 1 1 The W testing model ndash dynamic testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 28110
211 The W testing model ndash dynamic testing
28
2 1 1 The W testing model ndash static testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 29110
211 The W testing model static testing
29
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 30110
212 Software development models Waterfall
30
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 31110
212 Software development models Waterfall
Waterfall weaknesses
middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule
middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover
middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen
middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to
partition the system for delivery of pieces of the system
31
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 32110
p p yp
32
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 33110
p p yp
Rapid Prototype Model weaknesses
middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked
middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products
middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations
middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary
middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study
33
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 34110
34
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 35110
35
Incremental Model weaknesses
middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments
middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-
defined interfaces are required
middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 36110
36
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 37110
Spiral Model weaknesses
middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to
proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk
analysis and prototyping may be excessive
37
212 Software development models - Rational Unified Process
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 38110
38
213 Software development models ndash Testing life cycle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 39110
bull For each software activity there must be a corresponding testing activity
bull The objectives of the testing are specific to that ldquotestedrdquo activity
bull Plan analysis and design of a testing activity should be done during thecorresponding development activity
bull Review and inspection must be considered as part of testing activities
39
221 Test levels ndash Component testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 27110
27
Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled
Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled
Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level
2 1 1 The W testing model ndash dynamic testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 28110
211 The W testing model ndash dynamic testing
28
2 1 1 The W testing model ndash static testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 29110
211 The W testing model static testing
29
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 30110
212 Software development models Waterfall
30
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 31110
212 Software development models Waterfall
Waterfall weaknesses
middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule
middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover
middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen
middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to
partition the system for delivery of pieces of the system
31
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 32110
p p yp
32
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 33110
p p yp
Rapid Prototype Model weaknesses
middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked
middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products
middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations
middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary
middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study
33
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 34110
34
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 35110
35
Incremental Model weaknesses
middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments
middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-
defined interfaces are required
middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 36110
36
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 37110
Spiral Model weaknesses
middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to
proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk
analysis and prototyping may be excessive
37
212 Software development models - Rational Unified Process
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 38110
38
213 Software development models ndash Testing life cycle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 39110
bull For each software activity there must be a corresponding testing activity
bull The objectives of the testing are specific to that ldquotestedrdquo activity
bull Plan analysis and design of a testing activity should be done during thecorresponding development activity
bull Review and inspection must be considered as part of testing activities
39
221 Test levels ndash Component testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 28110
211 The W testing model ndash dynamic testing
28
2 1 1 The W testing model ndash static testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 29110
211 The W testing model static testing
29
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 30110
212 Software development models Waterfall
30
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 31110
212 Software development models Waterfall
Waterfall weaknesses
middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule
middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover
middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen
middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to
partition the system for delivery of pieces of the system
31
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 32110
p p yp
32
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 33110
p p yp
Rapid Prototype Model weaknesses
middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked
middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products
middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations
middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary
middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study
33
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 34110
34
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 35110
35
Incremental Model weaknesses
middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments
middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-
defined interfaces are required
middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 36110
36
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 37110
Spiral Model weaknesses
middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to
proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk
analysis and prototyping may be excessive
37
212 Software development models - Rational Unified Process
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 38110
38
213 Software development models ndash Testing life cycle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 39110
bull For each software activity there must be a corresponding testing activity
bull The objectives of the testing are specific to that ldquotestedrdquo activity
bull Plan analysis and design of a testing activity should be done during thecorresponding development activity
bull Review and inspection must be considered as part of testing activities
39
221 Test levels ndash Component testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 29110
211 The W testing model static testing
29
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 30110
212 Software development models Waterfall
30
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 31110
212 Software development models Waterfall
Waterfall weaknesses
middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule
middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover
middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen
middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to
partition the system for delivery of pieces of the system
31
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 32110
p p yp
32
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 33110
p p yp
Rapid Prototype Model weaknesses
middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked
middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products
middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations
middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary
middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study
33
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 34110
34
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 35110
35
Incremental Model weaknesses
middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments
middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-
defined interfaces are required
middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 36110
36
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 37110
Spiral Model weaknesses
middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to
proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk
analysis and prototyping may be excessive
37
212 Software development models - Rational Unified Process
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 38110
38
213 Software development models ndash Testing life cycle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 39110
bull For each software activity there must be a corresponding testing activity
bull The objectives of the testing are specific to that ldquotestedrdquo activity
bull Plan analysis and design of a testing activity should be done during thecorresponding development activity
bull Review and inspection must be considered as part of testing activities
39
221 Test levels ndash Component testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 30110
212 Software development models Waterfall
30
212 Software development models - Waterfall
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 31110
212 Software development models Waterfall
Waterfall weaknesses
middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule
middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover
middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen
middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to
partition the system for delivery of pieces of the system
31
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 32110
p p yp
32
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 33110
p p yp
Rapid Prototype Model weaknesses
middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked
middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products
middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations
middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary
middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study
33
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 34110
34
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 35110
35
Incremental Model weaknesses
middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments
middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-
defined interfaces are required
middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 36110
36
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 37110
Spiral Model weaknesses
middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to
proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk
analysis and prototyping may be excessive
37
212 Software development models - Rational Unified Process
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 38110
38
213 Software development models ndash Testing life cycle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 39110
bull For each software activity there must be a corresponding testing activity
bull The objectives of the testing are specific to that ldquotestedrdquo activity
bull Plan analysis and design of a testing activity should be done during thecorresponding development activity
bull Review and inspection must be considered as part of testing activities
39
221 Test levels ndash Component testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 31110
212 Software development models Waterfall
Waterfall weaknesses
middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule
middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover
middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen
middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to
partition the system for delivery of pieces of the system
31
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 32110
p p yp
32
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 33110
p p yp
Rapid Prototype Model weaknesses
middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked
middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products
middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations
middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary
middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study
33
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 34110
34
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 35110
35
Incremental Model weaknesses
middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments
middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-
defined interfaces are required
middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 36110
36
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 37110
Spiral Model weaknesses
middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to
proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk
analysis and prototyping may be excessive
37
212 Software development models - Rational Unified Process
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 38110
38
213 Software development models ndash Testing life cycle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 39110
bull For each software activity there must be a corresponding testing activity
bull The objectives of the testing are specific to that ldquotestedrdquo activity
bull Plan analysis and design of a testing activity should be done during thecorresponding development activity
bull Review and inspection must be considered as part of testing activities
39
221 Test levels ndash Component testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 32110
p p yp
32
212 Software development models - Rapid Prototype Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 33110
p p yp
Rapid Prototype Model weaknesses
middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked
middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products
middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations
middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary
middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study
33
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 34110
34
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 35110
35
Incremental Model weaknesses
middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments
middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-
defined interfaces are required
middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 36110
36
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 37110
Spiral Model weaknesses
middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to
proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk
analysis and prototyping may be excessive
37
212 Software development models - Rational Unified Process
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 38110
38
213 Software development models ndash Testing life cycle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 39110
bull For each software activity there must be a corresponding testing activity
bull The objectives of the testing are specific to that ldquotestedrdquo activity
bull Plan analysis and design of a testing activity should be done during thecorresponding development activity
bull Review and inspection must be considered as part of testing activities
39
221 Test levels ndash Component testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 33110
p p yp
Rapid Prototype Model weaknesses
middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked
middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products
middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations
middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary
middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study
33
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 34110
34
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 35110
35
Incremental Model weaknesses
middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments
middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-
defined interfaces are required
middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 36110
36
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 37110
Spiral Model weaknesses
middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to
proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk
analysis and prototyping may be excessive
37
212 Software development models - Rational Unified Process
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 38110
38
213 Software development models ndash Testing life cycle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 39110
bull For each software activity there must be a corresponding testing activity
bull The objectives of the testing are specific to that ldquotestedrdquo activity
bull Plan analysis and design of a testing activity should be done during thecorresponding development activity
bull Review and inspection must be considered as part of testing activities
39
221 Test levels ndash Component testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 34110
34
212 Software development models - Incremental Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 35110
35
Incremental Model weaknesses
middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments
middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-
defined interfaces are required
middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 36110
36
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 37110
Spiral Model weaknesses
middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to
proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk
analysis and prototyping may be excessive
37
212 Software development models - Rational Unified Process
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 38110
38
213 Software development models ndash Testing life cycle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 39110
bull For each software activity there must be a corresponding testing activity
bull The objectives of the testing are specific to that ldquotestedrdquo activity
bull Plan analysis and design of a testing activity should be done during thecorresponding development activity
bull Review and inspection must be considered as part of testing activities
39
221 Test levels ndash Component testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 35110
35
Incremental Model weaknesses
middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments
middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-
defined interfaces are required
middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 36110
36
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 37110
Spiral Model weaknesses
middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to
proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk
analysis and prototyping may be excessive
37
212 Software development models - Rational Unified Process
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 38110
38
213 Software development models ndash Testing life cycle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 39110
bull For each software activity there must be a corresponding testing activity
bull The objectives of the testing are specific to that ldquotestedrdquo activity
bull Plan analysis and design of a testing activity should be done during thecorresponding development activity
bull Review and inspection must be considered as part of testing activities
39
221 Test levels ndash Component testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 36110
36
212 Software development models - Spiral Model
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 37110
Spiral Model weaknesses
middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to
proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk
analysis and prototyping may be excessive
37
212 Software development models - Rational Unified Process
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 38110
38
213 Software development models ndash Testing life cycle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 39110
bull For each software activity there must be a corresponding testing activity
bull The objectives of the testing are specific to that ldquotestedrdquo activity
bull Plan analysis and design of a testing activity should be done during thecorresponding development activity
bull Review and inspection must be considered as part of testing activities
39
221 Test levels ndash Component testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 37110
Spiral Model weaknesses
middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to
proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk
analysis and prototyping may be excessive
37
212 Software development models - Rational Unified Process
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 38110
38
213 Software development models ndash Testing life cycle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 39110
bull For each software activity there must be a corresponding testing activity
bull The objectives of the testing are specific to that ldquotestedrdquo activity
bull Plan analysis and design of a testing activity should be done during thecorresponding development activity
bull Review and inspection must be considered as part of testing activities
39
221 Test levels ndash Component testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 38110
38
213 Software development models ndash Testing life cycle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 39110
bull For each software activity there must be a corresponding testing activity
bull The objectives of the testing are specific to that ldquotestedrdquo activity
bull Plan analysis and design of a testing activity should be done during thecorresponding development activity
bull Review and inspection must be considered as part of testing activities
39
221 Test levels ndash Component testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 39110
bull For each software activity there must be a corresponding testing activity
bull The objectives of the testing are specific to that ldquotestedrdquo activity
bull Plan analysis and design of a testing activity should be done during thecorresponding development activity
bull Review and inspection must be considered as part of testing activities
39
221 Test levels ndash Component testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 40110
40
bull Target single software modules components that are separately testable
bull Access to the code being tested is mandatory usually involves the programmer
bull May consist of
o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)
bull Test cases follow the low level specification of the module
bull Can be automated (test driven software development)
o Develop first test codeo Then write the code to be testedo Execute until pass
bull Also named Unit testing
bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing
222 Test levels ndash Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 41110
bull Target the interfaces between components and interfaces with other parts ofthe system
We focus on thedata
exchanged not on the tested functionalitiesbull Product software architecture understanding is critical
bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)
bull Test strategy may be bottom-up top-down or big-bang
41
222 Test levels ndash Component Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 42110
Component integration testing (done after Component testing)
bullLinking a few components to check that they communicate correctly
bullIteratively linking more components together
bullVerify that data is exchanged between the components as required
bullIncrease the number of components create amp test subsystems and finally the
complete systemDrivers and Stubs should be used when necessary
bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system
bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces
a called component42
222 Test levels ndash System Integration testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 43110
System integration testing (done after System or Acceptance testing)
Testing the integration of systems and packages testing interfaces to externalorganizations
We check the data exchanged between our system and other external systems
Additional difficulties
bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments
Approaches to access the external systems
bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment
43
223 Test levels ndash System testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 44110
System testing = The process of testing an integrated system to verify that it meets
specified requirementsbull Target the whole product (system) as defined in the scope document
bull Environment issues are critical
bull May consist of
o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)
bull Black box testing techniques may be used (ex business rule decision table)
bull Test strategy may be risk based
bull Test coverage is monitored
44
224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 45110
Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies
the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system
The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client
The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives
bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users
bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site
45
231 Test types ndash Functional testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 46110
Target Test the functionalities (features) of a product
bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios
bull Focused on checking the system against the specifications
bull Can be performed at all test levels
bull Considers the external behavior of the system
bull Black box design techniques will be used
bull
Security testing is part of functional testing related to the detection of threats
46
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 47110
232 Test types ndash Non-Functional testing - Usability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 48110
Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions
bullPeople selected from the potential users maybe involved to study how they use the system
bullA quick and focused beta-test may be acheap way of doing Usability testing
bullThere is no simple way to examine howpeople will use the system
bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate
48
232 Test types ndash Non-Functional testing - Instalability
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 49110
Instalability testing = The process of testing the installability of a softwareproductrdquo
bullDoes the installation work
bullHow easy is to install the system
bullDoes installation affect other software
bullDoes the environment affect the product
bullDoes it uninstall correctly
49
232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 50110
50
Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system
Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements
Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching
bullResponse timebullThroughputbullResources utilization
Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits
Endurance Test = a Load Test performed for a long time interval (week(s))
Volume Test = Testing where the system is subjected to large volumes of data
233 Test types ndash Structural testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 51110
bull Targeted to test
o internal structure (component)o architecture (system)
Uses only white box design techniquesbull Can be performed at all test levels
bull Used also to help measure the coverage ( of items being covered by tests)
bull Tool support is critical
51
234 Test types ndash Confirmation amp regression testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 52110
Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed
bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)
Regression testing = Re-testing of a previously tested program following
modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed
bull
Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)
52
24 Maintenance testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 53110
Maintenance testing = Testing the changes to an operational system or the
impact of a changed environment to an operational system
Done on an existing operational system triggered by modification retirement ormigration of the software
Include
bullRelease based changesbullCorrective changesbullDatabase upgrades
Regression testing is also involved
Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)
53
31 Reviews and the testing process
S i T i i f ifi i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 54110
Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)
Reviews
Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications
The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual
54
31 Reviews and the testing process
When to review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 55110
When to review
bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development
Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions
Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong
55
321 Phases of a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 56110
Formal review phases
bullPlanning define scope select participants allocate roles define entry amp
exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria
56
322 Roles in a formal review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 57110
The formal reviews can use the following predefined roles
bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates
various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting
57
323 Types of review
Informal review Technical Review
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 58110
58
Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit
WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding
Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards
InspectionFormal process based on checklists entry and
exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory
Follow-up processMain purpose find defects
324 Success factors for reviews
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 59110
bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull
Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists
59
33 Static analysis by tools
bullPerformed without executing the examined software but assisted by tools
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 60110
Performed without executing the examined software but assisted by tools
bullThe approach may be data flow or control flow basedbullBenefits
bull early defects detection
bull early warnings about unwanted code complexitybull detects missing links
bull improved maintainability of code and design
bull Typical defects discovered
bull reference to an un-initialized variable
bull never used variables
bull unreachable code
bull programming standards violations
bull security vulnerabilities
60
4 Test design techniques - glossary
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 61110
bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull
Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular
objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases
61
41 Test design ndash test development process
2 Develop test cases1 Identify test conditions
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 62110
62
middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned
middot Traceability matrix (Use cases x Test cases)should be maintained
3 Develop test procedures
middot Group test cases into execution schedulesmiddot Factors to be considered
a Prioritizationb Logical dependenciesc Regression tests
middot Traceability from test condition to thespecifications (requirements) is a must
middot Risk analysis is a best practice
InputsbullField ndash levelbullGroup ndash level
Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation
bullField action messagebullRecord row windowbullFile table screenbullDatabase
bullProduct statesbullBehavior rules
Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions
42 Test design ndash categories of test design techniques
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 63110
Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts
Each black box or white box test technique has
bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)
Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull
Experience based testerrsquos knowledge about the specific domain about thelikely defects is used
63
431 Black box techniques ndash equivalence partitioning
bull To minimize testing partition input (output) values into groups of equivalent
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 64110
64
g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value
If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one
above it Example Rule for hiring a person is second its age
0 ndash 15 = do not hire
16 ndash 17 = part time18 ndash 54 = full time
55 -- 99 = do not hire
Which are the valid equivalence classes And the invalid ones
Give examples of representative values
(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)
431 Black box techniques ndash all-pairs testing
In practice there are situations when a great number of combinations must be
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 65110
In practice there are situations when a great number of combinations must be
tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux
bullTest environment combinations
bull8 browsers
bull3 plug-ins
bull6 client operating systems
bull3 servers
bull3 server OS
bull1296 combinations
bull All-pairs testing is the solution tests a significant subset of variables pairs
65
432 Black box techniques ndash boundary value analysis
Boundaries = edges of the equivalence classes
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 66110
66
Boundaries edges of the equivalence classes
Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values
bullFirst identify the equivalence classes
bullSecond identify the boundaries of each equivalence class
bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units
bullFor the previous example
bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values
-1011415161718195455569899100
(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)
433 Black box techniques ndash decision tables
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 67110
67
bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state
(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)
433 Black box techniques ndash decision tables - example
a b c are they the edges of a triangle
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 68110
(however some additional test cases are needed)
68
434 Black box techniques ndash state transit ion tables
Allow the tester to interpret the system in term of
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 69110
p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions
bull Transition table used
(see Lee Copeland b2 chap 7)
69
434 Black box techniques ndash state transition tables - example
Ticket buy - web application
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 70110
Exercise Fill-in the transition table
70
435 Black box techniques ndash requirements based testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 71110
Best practices
bullValidate requirements (what) against objectives (why)
bullApply use cases against requirements
bullPerform ambiguity reviews
bullInvolve domain experts in requirements reviews
bullCreate cause-effect diagrams
bullCheck logical consistency of test scenarios
bullValidate test scenarios with domain experts and users
bullWalk through scenarios comparing with design documents
bullWalk through scenarios comparing with code
71
435 Black box techniques ndash scenario testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 72110
Good scenario attributesbullIs based on a real story
bullIs motivating for the tester
bullIs crediblebullInvolves an enough complex use of environment and data
bullIs easy to evaluate ( no need for external oracle)
How to create good test scenariosbullWrite down real-life stories
bullList possible users analyze their interests and objectives
bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features
bullWatch users using old versions of the system or an analog system
bullStudy complaints about other analog systems 72
435 Black box techniques ndash use case testing
Generating the Test Cases from the Use Cases
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 73110
Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up
Steps1 Identify the use-case
scenarios2 For each scenario identify one
or more test cases3 For each test case identify theconditions that will cause it toexecute
4 Complete the test case byadding data values(see example)
73
435 Black box techniques ndash Syntax testing
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 74110
Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent
The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols
Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)
float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9
Syntax testing is the only black box technique without a coverage metricassigned
74
44 White box techniques ndash Control flow
Modules of code are converted to graphs the paths through the graphs are analyzed and
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 75110
g p p g g p y
test cases are created from that analysis There are different levels of coverage Process Blocks
A process block is a sequenceof program statements thatexecute sequentially
No entry into the block ispermitted except at thebeginning No exit from the
block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially
75
Decision Point
A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits
Junction Point
A junction point is a point atwhich control flows jointogether
Example
(see Lee Copeland b2 chap10)
441 White box techniques ndash statement coverage
Statement coverage = Executed statements Total executable statements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 76110
Example
a
if (b)
c
d
In case b is TRUE executing the code will result in 100 statement coverage
76
441 White box techniques ndash statement coverage - exercise
Given the code
a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 77110
if (x)
b
if (y)
c
else
d
else
e
x T T F F
y T F T F
a a a a
b b
c
d
e e
77How many test cases are needed to get 100 statement coverage
442 White box techniques ndash branch amp decision coverage - glossary
Branch = a conditional transfer of control from a statement to any other statement
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 78110
OR
= an unconditional transfer of control from a statement to any otherstatement except the next statement
Branch coverage = executed branches total branches
Decision coverage = executed decisions outcomes total decisions
For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage
78
442 White box techniques ndash branch amp decision coverage - example
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 79110
Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10
Q1 What are the decision and branch coverage for (B1 B2B9)
Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)
Answers 1 16 210 2 56 910
79
442 White box techniques ndash LCSAJ coverage
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 80110
LCSAJ = Linear Code Sequence and Jump
Defined by a triple conventionally identified by line numbers in a sourcecode listing
bullthe start of the linear code sequencebullthe end of the linear code sequence
bullthe target line to which control flow is transferred
LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq
80
443 White box techniques ndash data flow coverage
Just as one would not feel confident about a program without executing every
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 81110
statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation
Data flow coverages
bullAll defs = Number of exercised definition-use pairs Number of variable definitions
bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs
bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs
bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs
bullBranch condition = Boolean operand values executed Total Boolean operandvalues
bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations
81(see Lee Copeland b2 chap11)
45 Exploratory testing
bull Exploratory testing = Concurrent test design test execution test logging
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 82110
and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around
bullThe project environment
bullQuality criteria defined for the project
bullElements of the product
bullRisk factors
82
46 Choosing test techniques
Factors used to choose
middot Product or system type middot Schedule constraints
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 83110
middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks
middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience
(additional materials Unit Test design exercises )
83
511 Test organization amp independence
Options independent team or not
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 84110
Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues
Minusesbull Risk of isolation from the development teambull Communication issues
middot Developers can loose the lsquoquality ownershiprsquo attribute
84
512 Tasks of the test leader
bullPlan estimates test effort collaborates with project manager
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 85110
bullElaborates the test strategy
bullInitiate test specification implementation execution
bullSet-up configuration management of test environment amp deliverables
bullMonitors and controls the execution of tests
bullChooses suitable test metrics
bullDecides if and to what degree to automate the tests
bullSelect tools
bullSchedule tests
bullPrepare summary test reports
bullEvaluate test measurements
85
512 Tasks of the tester
Test Designer
bull
Test Analyst
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 86110
86
Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability
Define testing environment details
Tester
bull Define test approach (procedure)bull
Write test casesbull Review test cases (peer review)bull Implement and execute tests
Record defects prepare defect reports
bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull
Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures
521-522-523 ndash Test planning
Test plan = A document describing the scope approach resources and schedule of intendedtest activities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 87110
It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning
It is a record of the test planning process
IEEE 829 Test plan contents
bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs
bullSchedulesbullRisks and contingenciesbullApprovals
bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested
bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria
87
521-522-523 ndash Test planningbull Determine scope
o Study project documents usedsoftware life-cycle specifications product
bull Refine plano Define roles detailedresponsibilities
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 88110
88
desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations
bull
Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify
mitigation actionsbull
Estimate testing effort determine costsdevelop schedule
o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders
bull
o Select test strategy test levelsTest strategy issues (alternatives)
bull Preventive approachbull
Reactive approachbull Risk-basedbull Model (standard) based
bull Choosing testing techniques (white
andor black box)o Select metrics to be used for defecttracking coverage monitoring
o Define entry and exit criteriabull Exit cr iteria
bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based
524 ndash Test estimation
Two approaches
b d i (hi i l d )
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 89110
bull based on metrics (historical data)bull made by domain experts
Testing effort depends onbull product characteristics (complexity specification)
bull development process (team skills tools time factors)
bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)
Time for confirmation testing and regression testing must beconsidered too
89
525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission
The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 90110
rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done
One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun
bullPreventative approaches where tests are designed as early as possible
bullReactive approaches where test design comes after the software or system has been produced
90
Or another taxonomy
Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-
based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative
Regression-averse
53 ndash Test progress monitor ing reporting amp control
C t l id tif d i l t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 91110
Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities
Possible corrective actions
bullAssign extra resource
bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria
Monitoring - Test metrics used
bullTest cases ( passed failed)
bullDefects (found fixedfound densitytrends)
bullTest Coverage ( executed Test cases)
Reporting
bullDefects remaining
bullCoverage metrics
bullIdentified risks
91
54 ndash Configuration management
IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to
identif and doc ment the f nctional and ph sical characteristics of a
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 92110
92
bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and
bullverify compliance with specified requirements
Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of
the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary
Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management
bullRelated to Version Control and Change Control
54 ndash Configuration management
Configuration Management activities
Configuration identification = selecting the configuration items for a system
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 93110
Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation
Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification
Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes
Configuration auditing = The function to check that the software productmatches the configuration items identified previously
93
54 ndash Configuration management
I T i C fi i M
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 94110
In Testing Configuration Management must
bullIdentify all test-ware items
bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle
bullSet and maintain the version of these itemsbullTrack the changes of these items
bullRelate test-ware items to other software development items inorder to maintain traceability
bullReference clearly all necessary documents in the test plansand test cases
94
55 ndash Risk amp Testing
Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 95110
bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)
The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help
95
56 ndash Incident management
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 96110
Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction
bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing
Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)
The incident reports raised against products defects are named also bugreports
96
56 ndash Incident management
Recommended Bug report format
bullDefect ID
Bug statuses
Issued ndash just been reported
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 97110
Defect IDbullComponent name and Build version
bullReported by and Date
bullError type
bullSeverity
bullPriority
bullSummary and detailed description
bullAttached documents
(Exercise )
Issued just been reportedOpened ndash programmer is working to solve-it
Fixed ndash programmer thinks thatrsquos repaired
Not solved ndash tester retested but the bug isnot solved
Deferred ndash programmer or PM decided topostpone the decision
Not-a-bug ndash programmer or testerdiscovered that it is not a defect
Closed ndash bug is solved and verified
97
611 ndash Test tool classification
bull Management of testing
bullTest management
bull Test execution
bullRecord and play
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 98110
Test managementbullRequirements management
bullBug tracking
bullConfiguration management
bull Static testing
bullReview support
bullStatic analysis
bullModeling
bull Test specification
bullTest design
bullTest data preparation
Record and playbullUnit test framework
bullResult comparators
bullCoverage measurement
bullSecurity
bull Performance and monitoring
bullDynamic analysis
bullLoad and stress testing
bullMonitoring
bull Specific application areas ( TTCN-3)
bull Other tools
98
612 ndash Tool support - Management of testing
bullTest management
bullManage testing activities
bullConfiguration management
bullIndividual support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 99110
Manage testing activitiesbullManage test-ware traceability
bullTest result reporting
bullTest metrics tools
bullRequirements management
bullChecking
bullTraceability
bullCoverage
bullBug tracking
Individual supportbullVersion and change control
bullBuilder
bullProject related
bullDepartment or company related
99
613 ndash Tool support - Static testing
bullReview support
bullProcess support
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 100110
bullCommunications support
bullTeam support
bullStatic analysis
bullCoding standards
bullWEB site structurebullMetrics
bullModeling bullSQL database management
100
614 ndash Tool support ndash Test specification
bullTest design
F i t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 101110
bullFrom requirements
bullFrom design models
bullTest stubs and driver generators
bullTest data preparation
101
615 ndash Tool support ndash Test execution and logging
bullRecord and play
bullScripting
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 102110
bullScripting
bullUnit test framework
bullTest harness frameworks
bullResult comparators
bullCoverage measurement
bullSecurity testing support
102
616 ndash Tool support ndash Performance and monitor ing
bullDynamic analysis
bullTime dependencies
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 103110
bullTime dependencies
bullMemory leaks
bullLoad testing
bullStress testing
bullMonitoring
Watch for possible mistakes
103
621 ndash Tool support ndash benefits
bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 104110
bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)
bullObjective assessment (eg static measures coverage and systembehavior)
bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)
104
621 ndash Tool support ndash risks
bullUnrealistic expectations for the tool (including functionality and ease ofuse)
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 105110
105
bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)
bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)
bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)
bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation
bullLack of stakeholders commitment for the implementation of a such tool
622 ndash Tool support ndash special considerations
bullTest execution tools
bullSignificant implementation effort
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 106110
Significant implementation effort
bullRecord and play tools are instable when changes occur
bullTechnical expertise is mandatory
bullPerformance testing tools
bullExpertise in design and results interpretation are mandatory
bullStatic analysis toolsbullLots of warnings generated
bullBuild management sensitive
bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical
bullTest tools future is much debated ( see hellip)
106
testing in Visual Studio Team System622 ndash Tool support ndash
bullDeveloper
bulluse Test DrivenDevelopment methods
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 107110
107
bullmanage Unit Testing
bullanalyze code coverage
bulluse code static analysisbulluse code profiler to handleperformance issues
bullTester
bullmanage test cases
bullmanage test suites
bullmanage manual testing
bullmanage bug trackingbullrecord play WEB tests
bullrun load tests
bullreport test results
622 ndash Tool support ndash testing in Agile distributed environment
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 108110
httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108
622 ndash Introducing a tool into an organization
bullTool selection process
bullIdentify requirements
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 109110
y q
bullIdentify constraints
bullCheck available tools on the market (feature evaluation)
bullEvaluate short list (feature comparison)
bullDemos
bullQuick pilotsbullSelect a tool
Note there are many free testing tools available some of them also online( wwwtestersdeskcom )
109
ISTQB Foundation Exam guidelines
K1 The candidates will recognize rememberand recall a term or concept
K2 The candidates can select the reasons orl ti f t t t l t d t th t i
bull 40 multiple (4) choice questionsbull 1 hour exam
bull Score gt= 65 (gt=26 good) t
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110
8112019 Curs Testare - Foundation
httpslidepdfcomreaderfullcurs-testare-foundation 110110
explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing
K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context
answers) to passbull 50 K1 30 K2 20 K3
bullChapter 1 - 7 questions
bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions
bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of
finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause
d) Testing is independently reviewing a system against its requirements 110