testing

Error

• People make errors. A good synonym is mistake. When people make mistakes while coding, we call these mistakes bugs.

• Errors tend to propagate; a requirements error may be magnified during design and amplified still more during coding.

Typographical error Misreading of

a specification

Misunderstanding of functionality of

a module

Error

Fault

A fault is the result of an error.

It is more precise to say that a fault is the representation of an error, where representation is the mode of expression, such as

– narrative text

– dataflow diagrams

– hierarchy charts– source code, and so on.

Failure

A failure occurs when a fault executes.

A particular fault may cause different failures, depending on how it has been exercised.

Incident When a failure occurs, it may or may not be readily apparent to

the users. An incident is the symptom(s) associated with a failure that alerts

the user to the occurrence of a failureTestA test is the act of exercising software with test cases. A Test

has two distinct goals– To find failures or– To demonstrate correct execution

Test

A test is the act of exercising software with test cases. A Test has two distinct goals– To find failures or

– To demonstrate correct execution

Test Case

• A test case has an identity and is associated with a program behavior.

• It has a set of inputs and a list of expected outputs.

Preconditions Actual Inputs

Inputs

Post Conditions Actual Outputs

Outputs

Typical Test Case

• Test Case ID:• Purpose:• Preconditions:• Inputs:• Expected Outputs:• Post conditions • Execution History

Date Result Version Run By

Testing Life Cycle

Requirement Specification

Fault Resolution

Design Fault

Isolation

CodingFault

Classification

Testing

Fault Incident

Fault

Error

Fault

Error

ErrorFix error

Testing Life Cycle

• The first three phases are Putting Bugs in• The testing phase is Finding Bugs• The last three phases are Getting Bugs OUT• The Fault Resolution Step is another opportunity

for errors

Verification and Validation

Verification

the process of determining whether or not the product of given phase fulfill the spec. from the previous phase

Validation

the process of evaluating the software at the end of the development to ensure the compliance with the requirements

Classification of V&V Activities

Most of the V&V activities can be classified as static

or dynamic

Static– Reviews

– program proving

– code reading

Dynamic• Testing by Executing the Code

Verification & Validation of Software

Insights from a Venn Diagram

• A very good view of testing is that it is the determination of the extent of program behavior that is both specified and implemented

• What can a tester do to make the region where these sets all intersect be as large as possible?

• How the test cases in the set T are identified?.

5 6

7

4 3

2

1

T

Test Cases

P

Program

S

Specificationfaults of omission

What if there are specified behaviors that have not been

programmed?

faults of commission

what if there are programmed

(implemented) behaviors that have not been specified?

What is Testing

“Software Testing is the Process of Exercising or evaluating a system by manual or automated means to verify that it satisfies specified requirements or to identify differences between expected and actual results.

IEEE Standard Glossary 1983

Identifying Test Cases

Fundamental approaches to identifying test cases 1. Functional Testing and

2. Structural Testing

Identifying Test Cases : Functional Testing

• Functional testing is based on the view that any program can be considered to be a function that maps values from its input domain to values in its output range

• This notion is commonly used in engineering, when systems are considered to be “black boxes”.

• This leads to the term Black Box Testing, in which the content (implementation) of a black box is not known, and the function of the black box is understood completely in terms of its inputs and outputs.

Identifying Test Cases : Functional Testing With the functional approach to test case identification, the

only information that is used is the specification of the software.

Advantages : – they are independent of how the software is implemented, so

if the implementation changes, the test cases are still useful, and

– test case development can occur in parallel with the implementation, thereby reducing overall project development interval.

Negative side:– there can be significant redundancies among test cases, and – this is compounded by the possibility of gaps of untested

software

Identifying Test Cases : Structural Testing The tester has access to the internal data structures and

algorithms including the code that implement these.

It is sometimes called White Box or even Clear Box Testing

Types of white box testing• Code coverage - creating tests to satisfy some criteria

of code coverage (e.g., the test designer can create tests to cause all statements in the program to be executed at least once)

• Data Flow Testing• Mutation testing methods

The Functional Versus Structural Testing Which is better - Functional Testing or Structural Testing

• Functional testing uses only the specification to identify test cases, while structural testing uses the program source code (implementation) as the basis of test case identification.

Neither approach, taken by itself, is sufficient.

• Consider program behaviors: if all specified behaviors have not been implemented, structural test cases will never be able to recognize this.

• Conversely, if the program implements behaviors that have not been specified, this will never be revealed by functional test cases.

• The quick answer is that both approaches are needed;

Error and Fault Taxonomies Process versus Product

• process refers to how we do something, and

• product is the end result of a process

SQA is more concerned with reducing errors endemic in the development process, while testing is more concerned with discovering faults in a product.

Faults can be classified in several ways:

• the development phase where the corresponding error occurred,

• the consequences of corresponding failures,

• difficulty to resolve,

• risk of no resolution, and so on.

Error and Fault Taxonomies

Fault taxonomy that distinguishes faults by the severity of their consequences [Beizer 84]

1. Mild Misspelled Word 2. Moderate Misleading or Redundant

Information 3. Annoying Truncated Names, bill for S$0.00 4. Disturbing Some Transactions not processed 5. Serious Lose a Transaction 6. Very Serious Incorrect transaction Execution 7. Extreme Frequent ‘ very serious’ errors 8. Intolerable Database Corruption 9. Catastrophic System Shutdown 10. Infectious Shutdown that spreads to other


Input/Output Fault

Type Instances

Input correct input not accepted

incorrect input accepted

description wrong or missing

parameters wrong or missing

Output wrong format

wrong result

correct result at wrong time (too early, too late)

incomplete or missing result

spurious result

spelling/grammar

cosmetic


Logic Faults

missing case(s)

duplicate case(s)

extreme condition neglected

misinterpretation

missing condition

extraneous condition(s)

test of wrong variable

incorrect loop iteration

wrong operator (e.g., < instead ≤)


Computational Faults

incorrect algorithm

missing computation

incorrect operand

incorrect operation

parenthesis error

insufficient precision (round-off, truncation)

wrong built-in function


Interface Faults

incorrect interrupt handling

I/O timing

call to wrong procedure

call to non-existent procedure

parameter mismatch (type, number)

incompatible types

superfluous inclusion


Data Faults

incorrect initialization incorrect data dimension

incorrect storage/access incorrect subscript

wrong flag/index value incorrect type

incorrect packing/unpacking incorrect data scope

wrong variable used sensor data out of limits

wrong data reference off by one

scaling or units error inconsistent data

Finding faults early

It is commonly believed that the earlier a defect is found the cheaper it is to fix it

The table shows the cost of fixing the defect depending on the stage it was found

Time Detected

Requirements Architecture Construction System

Test Post-

Release

Requirements 1× 3× 5–10× 10× 10–100×

Architecture - 1× 10× 15× 25–100× Time

Introduced Construction - - 1× 10× 10–25×

Testing in the Waterfall Model – V Model

The V-model is a software development model which can be presumed to be the extension of the waterfall model.

Instead of moving down in a linear way, the process steps are bent upwards after the coding phase, to form the typical V shape.

The V-Model demonstrates the relationship between each phase of the development life cycle and its associated phase of testing.

V- Model

Testing Principles

Testing Shows Presence of Defects

Testing can show that defects are present, but cannot prove that there are no defects. Testing reduces the probability of undiscovered defects remaining in the software but, even if no defects are found, it is not a proof of correctness.

Exhaustive Testing is Impossible

Testing everything (all combinations of inputs and preconditions) is not feasible except for trivial cases. Instead of exhaustive testing, we use risks and priorities to focus testing efforts.

Early Testing Testing activities should start as early as possible in the software or system development life cycle and should be focused on defined objectives.

Testing Principles

Defect Clustering A small number of modules contain most of the defects discovered during pre-release testing or show the most operational failures.

Pesticide Paradox

If the same tests are repeated over and over again, eventually the same set of test cases will no longer find any new bugs. Tb overcome this 'pesticide paradox', the test cases need to be regularly reviewed and revised, and new and different tests need to be written to exercise different parts of the software or system to potentially find more defects.

Testing is the Context Dependent

Testing is done differently in different contexts. For example, safety-critical software is tested differently from an . e-commerce site.

Absence of Error Fallacy

Finding and fixing defects does not help if the system built is unusable and does not fulfill the users' needs and expectations

Conclusion : Testing definition revisited

• The process consisting of all life cycle activities, both static and dynamic, concerned with planning, preparation and evaluation of software products and related work products to determine that they satisfy specified requirements, to demonstrate that they are fit for purpose and to detect defects.

What is Testing

“Software Testing is the Process of Exercising or evaluating a system by manual or automated means to verify that it satisfies specified requirements or to identify differences between expected and actual results.

IEEE Standard Glossary 1983

Practitioners' Views of Software Testing

• Checking programs against specifications • Finding bugs in programs • Determining user acceptability • Ensuring that a system is ready for use • Gaining confidence that it works • Showing that a program performs correctly • Demonstrating that errors are not present • Understanding the limits of performance • Learning what a system is not able to do • Evaluating the capabilities of a system • Verifying documentation • Convincing oneself that the job is finished and so forth.

Exploring Software Testing

• Good testing involves much more than just running the program a few times to see whether it works.

• Thorough analysis of the program helps us to test more systematically and more effectively.

For example, consider a program that add two numbers, entered by the user. Each number is of one or two digits. The program shall read the inputs and print the sum. Press <enter> after each number.

Exploring Software Testing

#include<stdio.h>

void Main()

{

int a, b, c;

scanf(“%2d”,&a);

scanf(“%2d”,&b);

c=a+b;

printf(“%2d\n”,c);

}

First cycle of Testing

Step I• Add 2 & 3• 5 will be displayed

Problem Reports After First Test

• The program worked in the sense that it accepted 2 &

3, and returned 5. But it still has problems.

• Design Errors:- No On screen Instructions. How do

we know what to do?

Instructions could easily be displayed on the screen.

• The errors are described on problem report form.

Problem Report Form Confidential

COMPANY’S NAME___________ PROBLEM REPORT #__________

PROGRAM______________ RELEASE_____________ VERSION_________

REPORT TYPE(1-6) ___ SEVERITY(1-3)__ ATTACHMENTS(Y/N)___

1. Coding error 1. Fatal If yes: describe:

2. Design Issue 2. Serious ______________________

3. Suggestion 3. Minor ______________________

4. Documentation

5. Hardware

6. Query

PROBLEM SUMMARY___________________________________________________

CAN WE REPRODUCE THE PROBLEM?(Y/N)__________

PROBLEM AND HOW REPRODUCE IT_____________________________________

_______________________________________________________________________

_

SUGGESTED FIX(optional)________________________________________________

_______________________________________________________________________

_

REPORTED BY_________________________________DATE___/____/___________

ITEMS BELOW ARE FOR USE ONLY BY THE DEVELOPMENT TEAM

FUNCTIONAL AREA___________________ ASSIGNED TO_______________

COMMENTS___________________________________________________________

STATUS(1-2)________________ PRIORITY(1-5)_______________

1. Open 2. Closed

RESOLUTION (1-9)___________ RESOLUTION VERSION___________

1. Pending 4. Deferred 7. Withdrawn by reporter

2. Fixed 5. As designed 8. Need more info

3. Irreproducible 6. Can’t be fixed 9. Disagree wit suggestion

RESOLVED BY____________________ DATE____/____/______

RESOLUTION TESTED BY__________________ DATE____/____/______

Problem Report Form

• Submit one problem report for each error. All errors could fit

on the same report but that is not a good idea.

• Problems that are grouped together might not be fixed at the

same time. The unfixed ones will be lost.

• If the developer wants to group them, he /she can sort the

reports himself/ herself.

Step II

• Make some notes about what else needs testing.

• Some of the notes may turn into formal test series that

is well-documented groups of tests that may be

probably used each time we test a new version of the

program.

TESTS OF VALID INPUT

Sr. No. Test Case Expected Results

Notes

1. 99 + 99 198 Largest pair of numbers the program can add

2. -99 +(-99) -198 The specification did not say we cannot use negative numbers.

TESTS OF VALID INPUT Contd…3. 99 + (-14) 85 A large first number might affect the

program’s interpretation of the second

4. -38 + 99 61 Check addition of a negative to a positive number.

5. 56 + 99 155 Large second number’s effect on first.

6. 9 + 9 18 9 is largest one digit number

7. 0 + 0 0 Programs often fail on zero.

8.

9.

0 + 23

-78 + 0

23

-78

0 should be tested in first and second entry position

Step III

• Check the invalid cases and see what happens.• The test series in previous table only covers valid

values.• Our next step is to create series like this for invalid

values.• Another important series would cover edited

numbers.

Step IV

• Always write down what we do and what happens

when we run test case.

• Try any test that seems promising, even if it is similar

to others that have already been run.

Step V

• Summarize what we know about the program and its

problems.

• To this point our thinking has been focused. We have

concentrated on specific issues, such as coming with boundary

conditions for valid inputs.

Summarize thoughts about the program

• The communication style of the program is very terse.• The program does not deal with two digit negative numbers. • The largest sum that it can handle is 198 and the smallest is 0.• The program accepts any character as valid input, until we

press <enter>.• If we type anything that is not a digit it gives a garbage value.

Second Cycle of Testing

Responses to the problems are given:-Design Issue:- No program title on screen Resolution:- Won’t be fixedDesign Issues:- No instruction on screen

Resolution:- fixedBug: - Cannot handle two digit negative numbers Resolution:- will add only one digit negative numbers.Bug:- if third digit is entered, it is taken as second number.

Resolution:- PendingBug:- Crashes when we enter non numbers

Resolution:- not a problem

Step I

• Review the responses of the problem reports

Step II

Review comments on problems that would not be fixed. They may suggest further tests.

The most serious problem in the program is terrible error handling.

Step II (contd..)

• The programmer does not intend to fix it. What can we do

about it?

• The single most effective tactic for getting a bug fixed is to

find test cases that make it appear so likely to occur under

such circumstances that absolutely no one would be willing to

tolerate it.

Remember:-

• A best tester is not the one who finds the most bugs

or who embarrasses the most programmers. The best

tester is the one who gets the most bugs fixed.

Software Testing – Self Assessment Test

• The program reads three integer values. The three values are interpreted as representing the lengths of the sides of a triangle. The program prints a message that states whether the triangle is Scalene, Isosceles, or Equilateral

• Write the set of Test Cases i.e. specific sets of data that you feel would adequately test this program


• Do you have a test case that represents a valid scalene triangle ? Test cases such as 1,2,3 and 2,5,10 is not correct

• Do you have a test case that represents a valid equilateral triangle

• Do you have a test case that represents a valid isosceles triangle ?

• Do you have at least 3 test cases that represent valid isosceles triangles such that you have tried all three permutations of two equal sides e.g. 3,3,4 3,4,3 and 4,3,3


• Do you have a test case in which one side has a zero values?• Do you have a test case in which one side has a negative

value?• Do you have a test case with three integers greater than zero

such that the sum of two of the numbers is equal to the third? (That is, if the program said that 1,2,3 represents a scalene triangle, it would contain a bug.)

• Do you have at least three test cases in above category such that you have tried all three permutations where the length of one side is equal to the sum of the lengths of the other two sides (e.g. 1,2,3; 1,3,2; and 3,1,2)?

• Do you have a test case with three integers greater than zero such that the sum of two of the numbers is less than the third (e.g. 1,2,4 or 12, 15, 30)?


• Do you have a least three test cases in above category such that you have tried all three permutations (e.g., 1,2,4; 1,4,2; and 4,1,2)?

• Do you have a test case in which all sides are 0 (I.e., 0,0,0)?

• Do you have at least one test case specifying no integer values?

• Do you have at least one test case specifying the wrong number of values (e.g., two, rather than three, integers)?

• For each test case, did you specify the expected output from the program in addition to the input values.

Sample Test Case

Test Case X Y Z Expected Output

1 100 100 1 Isosceles


3 100 100 200 Not a triangle


5 1 1 1 Equilateral


7 2 199 200 Scalene

8 100 199 200 Scalene


10 100 200 200 Isosceles11 100 2 200 Scalene12 199 100 100 Isosceles

Assuming Integer Greater than 0 and less than or equal to 200

Conclusion

• When one tests a program, one wants to add some value to the program.

• Adding value means raising the quality or reliability of the program.

• Raising the reliability of the program means finding and removing errors.

• Therefore, one should not test the program to show that it works; rather, one should start with the assumption that the program contains errors and then test the program to find as many of the errors as possible.

• Does testing implies that it is destructive process of trying to find the errors ?

Conclusion - What Testing Shows

errorserrors

requirements conformancerequirements conformance

performanceperformance

an indicationan indicationof qualityof quality

Conclusion

""Bugs lurk in corners and Bugs lurk in corners and congregate at boundaries…”congregate at boundaries…”

Boris BeizerBoris Beizer

OBJECTIVEOBJECTIVE

CRITERIACRITERIA

CONSTRAINTCONSTRAINT

to uncover errorsto uncover errors

in a complete mannerin a complete manner

with a minimum of effort and timewith a minimum of effort and time

Assignment

• You have been given a sample code for finding the minimum element in an array

• You are required to do the testing of the program so that faults, if any, are revealed

Sample Code - minimum

#include <stdio.h> #include<limits.h> #include<conio.h> void minimum(); void main() { minimum(); } void minimum() { int array[100]; int no; int i; int tmpdata; int minimum =INT_MAX; int minindex=0; printf("Enter the size of the array:"); scanf("%d", &no); for (i=0;i<no;i++) { printf("Enter A[%d]=", i+1); scanf("%d", &tmpdata); tmpdata=(tmpdata<0)? -tmpdata : tmpdata; array[i]=tmpdata; } i = 1; while (i<no-1) { if (minimum > array[i]) { minimum = array[i]; minindex=i; } i++; } printf("Minimum= %d at index %d\n", minimum, minindex+1); getch();

Papers to Study

• J.A. Whittaker, “What is software testing And why is it so hard?”, IEEE software, January / February, 70-77, 2000.

• J.L. LIONS et. al., “Report of the enquiry board constituted by Director – General of ESA for the identification of the causes of failure”, www.esrin.esa.it, July 19, Paris, 1996.

http://www.esrin.esa.it/

Thank You

Engineering

testing