Software Testing and Reliability Software Test Process

Software Testing and Reliability

Software Test Process

Aditya P. MathurPurdue UniversityAugust 12-16

@ Guidant CorporationMinneapolis/St Paul, MN

Graduate Assistants: Ramkumar NatarajanBaskar Sridharan

Last update: August 9, 2002

Software Testing and Reliability Aditya P. Mathur 2002 2

Learning Objectives

What is software life cycle?

How to mesh various types of testing into the development cycle?

What documents are generated for and as a result of testing?

What test-metrics needs to be collected and why?


References

Software Testing in the Real World, Edward Kit, Addison Wesley, 1995.


Life Cycle

There are many models of Life Cycle.

Software Development Cycle is a sequence of steps in which a software system is developed. We shall refer to this sequence as, simply, Life Cycle.

The waterfall model Iterative refinement model Incremental development model Spiral model Unified Process model


Testing and the Life Cycle

The key questions with which most process managers struggle are:

It is widely believed that testing is necessary in each phase of the life cycle regardless of which model is adopted.

How should the test process be “meshed” with the development cycle?

What tools and techniques should be used in each phase?

What data should be collected to evaluate the effectiveness of the test process?


Meshing the Test Process with the Life Cycle

Requirements specification

Designspecification

Code IntegrationSystem Integration

System Deployment

System Modification

Test the requirements

Test the modules Test the subsystems Test the system

Test for acceptance

Run regression Tests

Test the design


Test Tools and Techniques: Requirements Phase

Inspection and walkthroughs: Ensure completeness, unambiguity, precision, consistency, testability, and traceability. [Rational: Rose and Analyst Studio, I-Logix: Rhapsody]

Simulation: Use this to improve the understanding of the system to be built; where possible, work with the customer.

Generate test procedures, plans, and test cases: Develop procedures, plans and test cases to be used for testing each requirement. Identify what tools will be used in each phase, who will use these, and what data will be collected. Determine product release criteria.


Test Tools and Techniques: Design Phase

Inspection and walkthroughs [Rational: Rose]

Simulation: Check for the correctness of scenarios. [I-Logix: Rhapsody]

Refine test procedures and test cases: Use the artifacts created during design to generate additional test cases and refine or correct the existing ones.

Risk analysis: Use the application architecture to perform risk analysis. [Rational: ROOM]


Test Tools and Techniques: Coding Phase

Inspection and walkthroughs: Ensure that the code matches the design. If not then at leant one of the two needs to be modified to achieve consistency. [Rational: Rose]

Perform unit testing: Test each module thoroughly. [CleanScape: TestWise; Rational: Purify and Visual Test; AMC: CodeTest]


Test Tools and Techniques: Integration Phase

Perform integration testing: Test each subsystem thoroughly. [CleanScape: TestWise; Rational: Purify and Visual Test; Applied Microsystems: CodeTest; Windriver: Sniff+]


Test Tools and Techniques: System Integration

Perform system testing: Test the system to ensure correct functionality with respect to use cases, do performance, stress, and robustness tests. [CMU: Ballista; Telcordia: AETG; Applied Microsystems: CodeTest; Windriver: Diab RTA suite and Sniff+]


Test Tools and Techniques: System Changes

Perform regression testing: Test the modified system to ensure that existing functions perform as intended. Note that system testing for the added features ought to be performed. [CleanScape: Testwise for test minimization]


Test Tools and Techniques: In-target system

Perform system testing: Test the application for all functionality, its performance, and robustness while the application runs embedded in the target platform. [Advanced Microsystems: CodeTest; Home-grown test generators; Rational: Purify]


Test Tools and Techniques: Usability testing

Perform usability testing: Have a sample of the potential users test the application and the documents that accompany it. The goal is to ensure that (a) the documents are understandable, consistent, and correct and (b) the GUI satisfies the requirements of the potential users.

Usability testing can begin when the GUI is ready for use, i.e. developed and tested for correctness, but the remainder of the application code might not be.


Test Tools and Techniques: Security testing

Perform security and safety testing: Security and safety testing must be done when an application is accessible from the “outside” and might compromise the user’s privacy and safety,

The objective of security and safety testing is to demonstrate that privacy and safety cannot be compromised when the assumptions regarding the use of the application are met.

Security and safety testing is done with respect to the requirements for security and safety. Traditional testing techniques are useful but might have to be augmented with techniques specific to the application environment.


Responsibilities

Planning: (High level) Test team

Unit tests: Developer. (Who checks the developers work?)

Integration tests: Developer and/or test team depending on the complexity of the sub-system to be tested.

System level tests: Test team. These tests will generally require communication with the developers. This must be a part of the detail plan built at the beginning of the development cycle.

Acceptance and usability tests: Test team in collaboration with the end user (or a sample of the end user set).


Release Criteria

Units: Code coverage based. Example: all statements, decisions, and conditions must be covered.

Integration tests: Code coverage and function based. Low level code coverage might not be usable if the integration produces a “large” component.

System level tests: System reliability measured in terms of the number and type of failures detected.

Acceptance and usability tests: Determined by the user primarily based on correct functionality, robustness, performance, and usability.


Standards for Test Documents

SQAP: : Software Quality Assurance Plan, IEEE/ANSI, 1989 [Std 730-1989].

SVVP: Software Verification and Validation Plan, IEEE/ANSI, 1986 [Std 1012-1986]; one per SQAP.

VTP: Verification Test Plan, one per verification activity.

MVP: Master Validation Test Plan, IEEE/ANSI, 1983 [Std 829-1983]; one per SVVP.

DTP: Detailed Validation Test Plan, IEEE/ANSI, 1983 [Std 829-1983]; one or more per activity.


Standards for Test Documents (contd.)

TDS: : Test Design Specification, IEEE/ANSI, 1983 [Std 829-1983], one per DTP.

TCS: Test Case Specification IEEE/ANSI, 1983 [Std 829-1983]; one or more per TDS/TPS.

TPS: Test Procedure Specification, IEEE/ANSI, 1983 [Std 829-1983]; one or more per TDS.

TC: Test Case, one per TCS.


Structure of Standards for Test Documents

SQAP

SVVP

MTPVTP VTPVTP

DTP DTP DTP DTP

TDS TDS TDSTDS

TCS TCSTPS

TC TCTPS


Defect Tracking and Classification

Defect tracking and classification [Applied Innovation management: Bug/Defect Tracking Expert; defectx: defectX]

Defect estimation: Use inspection data for defect estimation; design metrics; past data on similar projects.

Defects open vs. closed

Cross-project and cross version comparison of the number of defects.

Cost of repairing a defect: time and manpower resources needed.


Why collect data?

Assists with improving the test process; correlations between other process related metrics, such as code coverage, and defects can be established.

Helps improve product quality.

Useful in setting up a reward system (no penalty please!).


Summary

Software development cycle

Test process and the software development cycle

Assignment of responsibilities

Standards

Metrics