Software Testing - Verification

Saturday, April 8, 2023Ch.19 - Verification & Validation 1

Software Testing: Verification and Validation

• Verification “Are we building the product right?”

• Validation “Are we building the right product?”

Barry Boehm, 1979


Verification and ValidationTechniques

• Static Techniques– Software Inspections

(against source code)

• Dynamic Techniques– Software Testing

(requires executable program)


Verification and ValidationStatic techniques

• Software Inspections– of requirements documents– of design documents (design reviews)– of source code (code reviews)– automated static analysis


Verification and ValidationDynamic techniques

• Software Testing– specification vs. implementation

• Defect testing [Ch.20]

– verifying non-functional requirements (e.g. performance; reliability)

• Statistical testing [Ch.21]

– automated dynamic analysis (if applicable)


Verification and ValidationGoals

• Establish that software is fit for purpose,not “bug-free”

• “Good enough” depends on:– Software function (critical nature?)– User expectations– Market

• competition, price


Testing vs. Debugging

• Verification and Validation– looking and categorizing existence of system

defects [example] [bug list]– “What?”

• Debugging– locating and correcting these defects– “Why?”

http://bugzilla.mozilla.org/show_bug.cgi?id=137551

http://bugzilla.mozilla.org/buglist.cgi?bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&bug_status=RESOLVED&email1=&emailtype1=substring&emailassigned_to1=1&email2=&emailtype2=substring&emailreporter2=1&bugidtype=incl


• Canned test runs to verify that new defects were not introduced during “debugging” session.

• Not exhaustive• Targeted to a particular interface

– components, sub-systems, integrated system

• Different levels (lengths) of regression tests– Targeted regressions

Regression Testing


• Planning should begin right after requirements specification– Acceptance tests can be written then

• System, sub-system tests can be written against designs

The Test Plan


The Test Plan

Requirementsspecification

Systemspecification

Systemdesign

Detaileddesign

Module andunit codeand tess

Sub-systemintegrationtest plan

Systemintegrationtest plan

Acceptancetest plan

ServiceAcceptance

testSystem

integration testSub-system

integration test


Software Inspections (code reviews)

• >60% of program errors can be detected in code review [Fagan86]

• >90% if more formal approach used (e.g. “Cleanroom” process) [Mills87](We’ll talk about Cleanroom later)



• Why are reviews more effective for finding defects in systems/subsystems (i.e., before acceptance testing)?– Bugs are often masked by other bugs– Leverage domain/programming knowledge

• inspectors are skilled programmers

• Common practice: code reviews and then acceptance testing– reviews can also help with development of tests



• Sample procedure:– Announce review meeting in advance (a week?)– Provide design document, implementation

overview, and pointer to code– Reviewers read code (and make notes) in

advance of meeting– During meeting, directives recorded by Scribe– Testers/documenters attend too


Automated Static Analysis

• CASE tools that catch program curiosities that are usually indicative of bugs:

– unreachable code– uninitialized variables– unreferenced variables

• Programming language dependent– e.g., LINT (for C)


Automated Dynamic Analysis

• Tools which do bookkeeping while program is run/tested.

• Can find some dynamic problems that compiler cannot catch (depends on language…)

• C/C++ tools: Purify, BoundsChecker

Documents

Software Testing - Verification