Software Reliability and Testing - Summer semester 2013 ... · Software Reliability and Testing...

Software Reliabilityand Testing

Barbara Russo

Outline

Testing andreliability

TestingSoftware Reliability and Testing

Summer semester 2013/2014Fundamentals of Testing

Barbara Russo

27 Feb 2014

1 / 19

Software Reliabilityand Testing

Barbara Russo

Outline

Testing andreliability

Testing

Table of Contents

1 Testing and reliability

2 Testing

2 / 19

Software Reliabilityand Testing

Barbara Russo

Outline

Testing andreliability

Testing

Testing and reliability

3 / 19

Software Reliabilityand Testing

Barbara Russo

Outline

Testing andreliability

Testing

Testing

According to IEEE definition:Software testing is the process of analyzing a software itemto detect the differences between existing and requiredconditions (that is, bugs) and to evaluate the features of thesoftware itemIn practice:Testing is the process of executing a program with the intentof finding an error

4 / 19

Software Reliabilityand Testing

Barbara Russo

Outline

Testing andreliability

Testing

Reliability

According to IEEE definition:Reliability is the ability of a system or component to performits required functions under stated conditions for a specifiedperiod of time

5 / 19

Software Reliabilityand Testing

Barbara Russo

Outline

Testing andreliability

Testing

Testing

Exhaustive testing is not feasible:if a failure is detected then the software is a failure software,but this does not imply that if no failure has been detectedthe software is correct

6 / 19

Software Reliabilityand Testing

Barbara Russo

Outline

Testing andreliability

Testing

Testing

Measuring and evaluating a software process may lead to theonly notion of “good” process we can define:a “good” process is correlated with the probability that nofailures occur in a given interval of time → Softwarereliability

7 / 19

Software Reliabilityand Testing

Barbara Russo

Outline

Testing andreliability

Testing

Testing

• Cost of software failures surpasses the cost of testing• Expenses for debugging and recall of product• Loss of customers? faith in company

• Effective test building• Uncovering as many defects as possible with minimum

amount of time and effort

• Is all of this feasible?

8 / 19

Software Reliabilityand Testing

Barbara Russo

Outline

Testing andreliability

Testing

Basic Questions

• When do testing start? When is it complete?

• What particular techniques should be applied duringsoftware development to get acceptable quality atacceptable cost?

• How can we assess the readiness of a product forrelease?

• How can we control the quality of a product to release?

• How can the development process itself be improvedover the course of current and future projects toimprove products and make verification morecost-effective?

9 / 19

Software Reliabilityand Testing

Barbara Russo

Outline

Testing andreliability

Testing

Basic Questions

• When do testing start? When is it complete?

• What particular techniques should be applied duringsoftware development to get acceptable quality atacceptable cost?

• How can we assess the readiness of a product forrelease?

• How can we control the quality of a product to release?

• How can the development process itself be improvedover the course of current and future projects toimprove products and make verification morecost-effective?

9 / 19

Software Reliabilityand Testing

Barbara Russo

Outline

Testing andreliability

Testing

Basic Questions

• When do testing start? When is it complete?

• What particular techniques should be applied duringsoftware development to get acceptable quality atacceptable cost?

• How can we assess the readiness of a product forrelease?

• How can we control the quality of a product to release?

• How can the development process itself be improvedover the course of current and future projects toimprove products and make verification morecost-effective?

9 / 19

Software Reliabilityand Testing

Barbara Russo

Outline

Testing andreliability

Testing

Basic Questions

• When do testing start? When is it complete?

• What particular techniques should be applied duringsoftware development to get acceptable quality atacceptable cost?

• How can we assess the readiness of a product forrelease?

• How can we control the quality of a product to release?

• How can the development process itself be improvedover the course of current and future projects toimprove products and make verification morecost-effective?

9 / 19

Software Reliabilityand Testing

Barbara Russo

Outline

Testing andreliability

Testing

Basic Questions

• When do testing start? When is it complete?

• What particular techniques should be applied duringsoftware development to get acceptable quality atacceptable cost?

• How can we assess the readiness of a product forrelease?

• How can we control the quality of a product to release?

• How can the development process itself be improvedover the course of current and future projects toimprove products and make verification morecost-effective?

9 / 19

Software Reliabilityand Testing

Barbara Russo

Outline

Testing andreliability

Testing

When do testing start? When is it complete?

• Testing activities are spread all over the developmentprocess

• There are several techniques at different stages of theprocess

• Different techniques involve different stakeholders

10 / 19

Software Reliabilityand Testing

Barbara Russo

Outline

Testing andreliability

Testing

When do testing start? When is it complete?

Figure : Main analysis and testing activities through the softwarelife cycle. Source: Pezze and Young (reference book)

11 / 19

Software Reliabilityand Testing

Barbara Russo

Outline

Testing andreliability

Testing

What techniques should be applied duringsoftware development?

• Feasibility study to understand and select appropriatetechniques

• Verification to lead development according torequirements

• Validation to check product against users’ expectiations

12 / 19

Software Reliabilityand Testing

Barbara Russo

Outline

Testing andreliability

Testing

What techniques should be applied duringsoftware development?

• Techniques depend on quality, cost, scheduling,resources

• Not “one technique”• Effectiveness for different classes of fault to capture.• Applicability at different stages of the project (early

stage artefacts might not be tested with automatedtools)

• Differences in purpose. Techniques can have differentgoals (to understand the coverage or to detect faults)

• Trade-off cost and assurance, e..g, focus on key fewproperties

13 / 19

Software Reliabilityand Testing

Barbara Russo

Outline

Testing andreliability

Testing

What techniques should be applied duringsoftware development?

• Feasibility study includes• Tentative architectural design to modularise work and

identify properties that can be verified in differentsubsystems

• Draft plan breaking projects into incremental deliveriespreliminary decisions about test and analysis techniques

14 / 19

Software Reliabilityand Testing

Barbara Russo

Outline

Testing andreliability

Testing

How can we assess the readiness of a product forrelease?

Dependability properties:

• Availability

• Mean Time Between Failures

• Reliability

15 / 19

Software Reliabilityand Testing

Barbara Russo

Outline

Testing andreliability

Testing

How can we assess the readiness of a product forrelease?

Examples of dependability measures:

• Availability: daily down time

• Mean Time Between Failures (MTBF): days betweentwo consecutive failures

• Reliability: number of functionalities that correctlyoperate

16 / 19

Software Reliabilityand Testing

Barbara Russo

Outline

Testing andreliability

Testing

How can we assess the readiness of a product forrelease?

How:

• alpha testing: performed by users in a controlledenvironment. Capture operational profiles decide by theorganisation.

• beta testing: performed by users in a their ownenvironment. Capture different operational profiles

17 / 19

Software Reliabilityand Testing

Barbara Russo

Outline

Testing andreliability

Testing

How can we control the quality of successivereleases?

• Major revisions also called “point releases.” The fullquality process is repeated including beta testing andregression testing.

• Small revisions also called “patch releases.” Revisionrelated to incorporate solution to fault fixing. Fasttesting often automated is foreseen. Subset ofregression testing.

• in Open Source Software we might have “milestones,”“release candidates,” and “major releases”

18 / 19

Software Reliabilityand Testing

Barbara Russo

Outline

Testing andreliability

Testing

How can we control the quality of successivereleases?

19 / 19

Software Reliabilityand Testing

Barbara Russo

Outline

Testing andreliability

Testing

How can the development process itself beimproved over the course of current and futureprojects?

• Data collection across projects for modelling andprediction, e.g., faults and their severity or priority

• Personal Software Process (Humphrey)

20 / 19