Software Quality Assurance & Testing

Mistake in coding is called error ,Error found by tester is called defect, Defect accepted by development team then it is called bug ,Product does not meet the requirements then it Is failure.”

Importance of Test There are hundreds of stories about failures of

computer systems that have been attributed to errors in software

There are many reasons why systems fail but the issue that stands out the most is the lack of adequate testing.

Example Pepsi - $42 Billion Error Philippines -1992Due to a software error, 800,000 bottle caps were produced with number 349 instead of one. It was equivalent to $42 billion in prize money instead of $40,0000

Testing Levels

Testing Goals Based on Test Process MaturityBezier Testing Levels

Level 0 There is no difference between testing and debugging Level 1 The purpose of testing is to show the correctness(software works).Level 2 The purpose of testing is to show that the software doesn’t work.Level 3 The purpose of testing is not to prove anything specific, but to reduce the risk of using the software.Level 4 Testing is a mental discipline that helps all IT professionals develop higher quality software.

Bezier Testing Levels Level 0 is the view that testing is the same as debugging.In Level 1 testing, the purpose is to show correctnessIn Level 2 testing, the purpose is to show failuresLevel 3 testing shows the presence , not absence or

failure. If we use software, we expose some risk. The risk may be small and unimportant, or the risk may be great and the consequences catastrophic, but risk is always there. In level 3 testing, both testers and developers work together to reduce risk.

Bezier Testing Levels

Once the testers and developers are on the same “team,” an organization can progress to real Level 4 testing.

Level 4 testing means that the purpose of testing is to improve the ability of the developers to produce high quality software.

Debugging and Testing

Debugging is done in the development phase by the developerIn debugging phase identified bugs are fixed

Testing is done in the tester phase by the tester.In testing, locating and identifying of the bugs are

included.

Software Bug

A software bug is an error, flaw, mistake, failure, or fault in a computer program or system

Bug is terminology of TesterSoftare bug produces an incorrect or

unexpected result, or causes it to behave in unintended ways

Software Fault

A static (physical)defect*, imperfection, flaw in the software.

short between wiresbreak in transistorinfinite program loop

A fault means that there is a problem within the code of a program which causes it to behave incorrectly.

*defect: Mismatch between the requirements

Software Error

Error is an incorrect internal state that is the demostration of some fault.

In other words; Error is a deviation from correctness or accuracy.

Example Suppose a line is physically shortened to 0. (there is a

fault). As long as the value on line is supposed to be 0, there is no error.

Errors are usually associated with incorrect values inthe system state.

Software FailureA failure means that the program has not performed

as expected, and the actual performance has not met the minimum standards for success.

External, incorrect behavior with respect to the requirements or other description of the expected behavior

ExampleSuppose a circuit controls a lamp (0 = turn off, 1 = turn

on) and the output is physically shortened to 0 (there is a fault).

As long as the user wants the lamp off, there is no failure.

eBay Crash

eBay in 1999 was a very large Internet auction house

–top 10 Internet business – Market value of $22 billion – 3.8 million users as of March 1999 – Access allowed 24 hours 7 days a weekIn June 6, 1999 eBay system was unavailable for 22

hours with problems ongoing for several daysStock drops by 6.5%, $billion lost revenues Problems blamed on Sun server software

Ariane 5 Rocket Crash (June 4, 1996)

Ariane 5 rocket exploided 37 seconds after lift-off onError was due to software bug Conversion of a 64-bit floating point number to a

16-bit integer resulted in an overflowDue to the overflow, the computer cleared its memory

Ariane 5 interpreted the memory dump as aninstruction to its rocket nozzles Testing of full system under actual conditions notdone due to budget limits Estimated cost: 60 million $

Nine Causes of Software Errors

1. Faulty requirements definition2. Client developer communication failures3. Deliberate deviations from software requirements4. Logical design errors5. Coding errors6. Noncompliance with documentation and coding instructions7. Shortcomings of the testing process8. User interface and procedure errors9. Documentation errors

Example: Failure & Fault & Error Consider a medical doctor making a diagnosis for a patient. The patient enters the doctor’s office with a list

of failures (that is, symptoms). The doctor then must discover the fault, or

root cause of the symptom. To aid in the diagnosis, a doctor may order tests that look for anomalous internal conditions. In our terminology, these anomalous internal

conditions correspond to errors.

Cause-and-effect relationship

Faults can result in errors. Errors can lead to system failuresErrors are the effect of faults. Failures are the effect of errorsBug in a program is a fault.

Possible incorrect values caused by this bug is an error.

Possible crush of the operating system is a failure

Cause & Effect Relationship

Origins of Faults

Specification Mistakes – incorrect algorithms, incorrectly specified requirements (timing, power, environmental) Implementation Mistakes – poor design, software coding mistakes, Component Defects – manufacturing imperfections, random device defects, components wear-outs External Factors – operator mistakes, radiation, lightning,

Program Exampleclass numZero { public static int numZero (int[] x) {// if x == null throw NullPointerException// else return the number of occurrences of 0 in x int count = 0; for (int i = 1; i < x.length; i++) if (x[i] == 0) count++;return count; } }

The Fault in the Example The fault in this program is that it starts looking

for zeroes at index 1 instead of index 0, as is necessary for arrays

First Input: numZero ([2, 7, 0]) correctly evaluates to 1,Second Input: numZero ([0, 7, 2]) incorrectly evaluates to 0.In both of these cases the fault is executed. Both of these cases result in an error, Only the second case results in failure.

Error States in the Example To understand the error states, we need to identify the state for the program.The state for numZero consists of values for the

variables x, count , i and the program counter (PC).

The state at the if statement on the first iteration of the loop is

( x = [2, 7, 0], count = 0, i = 1, PC = if)This state is in error, because the value of i

should be 0 on the first iteration.

Error States in the Example (for the first input)

HoweverThe value of count is correctThe error state does not affect the output, The software does not fail. Finally: A state is in error if it is not the expected

state, even if all of the values in the state are acceptable.

Error States in the Example (for the second input)

For the second input: The state for numZero consists of values for the

variables x, count , i and the program counter (PC).

The state is (x = [0, 7, 2], count = 0, i = 1, PC = if)The error affects the variable count The error is present in the return value of the

method. The failure results.

Distinguish Testing from Debugging

The definitions of fault and failure allow us to distinguish testing from debugging.

Big difference is that debugging is conducted by a programmer and the programmer fix the errors during debugging phase.

Tester never fixes the errors, but rather find them and return to programmer.

Testing versus Debugging

Testing activity is carried down by a team of testers, in order to find the defect in the software.

Test engineers run their tests on the piece of software and if they encounter any defect (i.e. actual results don't match expected results), they report it to the development team.

Testers also have to report at what point the defect occurred and what happened due the occurrence of that defect.

All this information will be used by development team to DEBUG the defect.

Testing versus Debugging

Debugging is the activity which is carried out by the developer. After getting the test report from the testing team about defect(s),

the developer tries to find the cause of the defect. He has to go through lines of code and find which part of

code in causing that defect. After finding out the bug, he tries to modify that portion of code

and then he rechecks if the defect has been finally removed. After fixing the bug, developers send the software back to testers.

What is Software Quality ?

According to the IEEE Software Quality is:The degree to which a system, component, or

process meets specified requirements.The degree to which a system, component, or

process meets customer or user needs or expectations.

Importance of Software Quality Software is a major component of computer systems (about 80% of

the cost) – used for – communication (e.g. phone system, email system) – health monitoring, – transportation (e.g. automobile, aeronautics), – economic exchanges (e.g. ecommerce), – entertainment, – etc. Software defects are extremely costly in term of – money – reputation – loss of life

Software Quality Factors

Software Quality FactorsCorrectness – accuracy, completeness of required output –datedness, availability of the informationReliability – maximum failure rate Efficiency – resources needed to perform software function Integrity – software system security, access rights Usability – ability to learn, perform required task

Software Quality Factors

Maintainability – effort to identify and fix software failures (modularity, documentation, etc) Flexibility – degree of adaptability (to new customers, tasks, etc)Testability – support for testing (e.g. log files, automatic diagnostics, etc)

Software Quality Factors

Portability – adaptation to other environments (hardware, software)Reusability – use of software components for other projects Interoperability – ability to interface with other components/systems

What is Software Quality Assurance?

According to the IEEE Software quality assurance is: A planned and systematic pattern of all actions

necessary to provide adequate confidence that an itemor product conforms to established technicalrequirements.

A set of activities designed to evaluate the process by which the products are developed or manufactured.

Contrast with: quality control.

Software Quality Assurance

Verification– are we building the product right ?– performed at the end of a phase to ensure that requirements established during previous phase have been metValidation– are we building the right product ?– performed at the end of the developmentprocess to ensure compliance with productrequirements

Three General Principles of Quality Assurance

Know what you are doingKnow what you should be doingKnow how to measure the difference

Verification & Validation

Verification: The process of determining whether the products of a given phase of the software development process fulfill the requirements established during the previous phase.

Validation: The process of evaluating software at the end of software development to ensure compliance with intended usage.

Difference Between Verification &Validation

Verification is preventing mechanism to detect possible failures before the testing begin. It involves reviews, meetings, evaluating

documents, plans, code, inspections, specifications etc.

Validation occurs after verification and it's the actual testing to find defects against the functionality or the specifications

The Difference between Verification &Validation

Verification is usually a more technical activity that uses knowledge about the individual software artifacts, requirements, and specifications.

Validation usually depends on domain knowledge; that is, knowledge of the application for which the software is written. For example, validation of software for an airplane

requires knowledge from aerospace engineers and pilots.