Software Testing 1

LOGO

Nguyễn Đạt ThôngFaculty of Mathematics and InformaticsUniversity of Science, VNU-HCM

Software TestingSoftware Testing

Course

�Name: Software Testing 1�Lecturer: Nguyễn Đạt Thông�Credit: 3�Duration:

� Presentation: 13 weeks� On-the-fly practice: 02 weeks� Lab practice: 10 weeks

�Prerequisites:� Software Engineering

Nguyễn Đạt Thông Software Testing 2

Content

1• Introduction

2• Test Principles

3• Test Types

4• Test Techniques

5• Test Methodologies

6• Test Automation

7• Test Management

8• Test Tools

9• Test a Software Project

3Software TestingNguyễn Đạt Thông

Practice

Test Analysis

Test Case

Test Report

Tester

Case

Bug Report

Bug Tracking

Report


Evaluate

�Mid-term exam: 30%� Type: Writing� Time: Up to chapter 6

�Lab practice: 30%� Type: On lab� Time: 15th week

�Final exam: 50%� Type: Multiple choice� Time: (Depend)


Reference

1. Internet Source. Keyword:Software Testing, Test Methodology, Test Principles, Test Tool, Test Technique.

2. Dorothy Graham, Erik van Veenendaal, 2. Dorothy Graham, Erik van Veenendaal, Isabel Evans, Rex Black“Foundations of Software Testing”

3. Software Testing Help http://www.softwaretestinghelp.com

4. The UML Testing Profile http://www.omg.org


Reference (2)

6. Wikipedia, the free encyclopedia http://en.wikipedia.org/wiki


Introduction

Chapter 1


Content

1• Introduction

2• Test Principles

3• Test Types

4• Test Techniques

5• Test Methodologies

6• Test Automation

7• Test Management

8• Test Tools

9• Test a Software Project


Introduction

�What is a bug?�Quality and Reliability�Why is testing necessary?�Purposes of testing�No bugs found…�How much testing is enough?�Test approaches�Which factors influence test?�Independence in test�Who are Testers?


What is a bug?

�Error: a human action that produces an incorrect result

�Fault: a manifestation of an error in software� also known as a defect or bug� also known as a defect or bug� if executed, a fault may cause a failure

�Failure: deviation of the software from its expected delivery or service � (found defect)


What is a bug? (2)

A person makes

an error...

… that creates abug in the


bug in thesoftware

...

… that can causea failure

in operation

Quality and Reliability

�Software quality:� Reliability: the probability that software will not

cause the failure of the system for a specified time under specified conditions.

� Efficiency: the ability to ensure high � Efficiency: the ability to ensure high performance once the application is in run-time mode.

� Security: the ability to encounter critical vulnerabilities that damage the business.

� Maintainability: includes adaptability, portability and transferability

Software Testing 13Nguyễn Đạt Thông

Quality and Reliability (2)

�The less failures system causes, the more reliable the system is.

�No system is bug-free. So a system can be reliable but still have bugs.

�Bugs occur in software since:�Bugs occur in software since:� Software is written by human beings

• Know something, but not everything• Have skills, but are not perfect• Make mistakes

� Software is developed under increasing pressure and strict deadlines


Why is testing necessary?

�Software failures cost:� Huge sums

• Ariane 5: $7,000,000,000• Mariner space probe to Venus: $250,000,000• American Airline: $50,000,000• American Airline: $50,000,000

� Very little or nothing at all• Minor inconvenience• No visible

�Software is not “linear”:� Small input may have very large effect


Why is testing necessary? (2)

�Software failures can cause death or injury:� Radiation treatment kills patients� Train driver killed� Aircraft crashes (Airbus & Korean Airlines)� Aircraft crashes (Airbus & Korean Airlines)� Bank system overdraft letters cause

suicide


Why is testing necessary? (3)

�Testing is necessary:� because software is likely to have bugs� because failures can be very expensive� to learn about the reliability of the software� to avoid being sued by customers� to avoid being sued by customers� to stay in business

�Testing is not:� to prove that the software has no bugs� to fill the time before the release date� because testing is included in the project plan


Purposes of testing

�Purposes of testing are to:� find bugs/faults � build confidence� demonstrate conformance to requirements� reduce costs� show system meets user needs� assess the software quality� improve the software quality: reliability,

efficiency, security, usability, maintainability, reusability, testability, etc.


No bugs found…

ConfidenceBugs found

�No bugs found = confidence?


Time

No bugs found… (2)

ManyBugs

FewBugs

You thinkyou are here

High

Low High


FewBugs

FewBugs

You may be here

TestQuality

Low

Software Quality

Low High

How much testing is enough?

system has20 screens

Average 4 menus3 options / menu

Average: 10 fields / screen2 types input / field(date as Jan 3 or 3/1)(number as integer or decimal)

� Total for 'exhaustive' testing:

�� 20 x 4 x 3 x 10 x 2 x 100 = 20 x 4 x 3 x 10 x 2 x 100 = 480,000 tests480,000 tests

� 1 sec/test-> 8000 mins ~ 133 hrs ~ 17.7 days

(not counting finger trouble, bugs or retest)

� 10 secs-> 34 wks, 1 min-> 4 yrs, 10 min-> 40 yrs


(number as integer or decimal)Around 100 possible values

We cannot test everything

How much testing is enough? (2)

�Exhaustive testing:� is exercising all combinations of inputs and

preconditions� consumes impractical amount of time

�We do not run an exhaustive test�We do not run an exhaustive test� not enough resources� not enough time� not necessary



�it’s never enough�when you have done what you planned�when your customer/user is happy�when you have proved that the system

works correctlyworks correctly�when you are confident that the system

works correctly�it depends on the risks for your system



�So little time vs. So much to test:� test time will always be limited� base on RISK, we have to determine:

• what to test first• what to test most• what to test most• how thoroughly to test each item• what not to test (this time)

� we have to allocate the time available for testing by prioritizing testing



�Other factors that influence testing� contractual requirements� legal requirements� industry-specific requirements

• e.g. pharmaceutical industry (FDA), compiler standard tests, safety-critical or safety-related such as railroad switching, air traffic control



�Most important principles:

It is difficult to determinehow much testing is enoughbut it is not impossible


Prioritize testsso that,

whenever you stop testing,you have done the best testing

in the time available.

Test approaches

�Traditional testing approach� Show that the system:

• does what it should• doesn't do what it shouldn't

� Easy test cases� Easy test cases


Goal: show working

Success: system works

Result: bugs left in

Test approaches (2)

�A better testing approach� Show that the system:

• does what it shouldn't• doesn't do what it should

� Difficult test cases� Difficult test cases


Goal: find bugs

Success: system fails

Result: fewer bugs left in

Test approaches (3)

�The testing paradox� Purpose of testing is to destroy confidence:

• Purpose of testing is to try to find bugs• Finding bugs destroys confidence

� Purpose of testing is to build confidence� Purpose of testing is to build confidence


The best way to build confidence

is to try to destroy it

Independence in test

�Test your own work?� find 30% - 50% of your own bugs� same assumptions and thought processes� see what you meant or want to see, not

what is therewhat is there� emotional attachment

• don’t want to find bugs• actively want NOT to find bugs

�Independence in test is very important.


Independence in test (2)

�Levels of independence� Tests designed by the person who wrote

the software� Tests designed by a different person� Tests designed by someone from a � Tests designed by someone from a

different department or team (e.g. test team)

� Tests designed by someone from a different organisation (e.g. agency)

� Tests generated by a tool (low quality tests?)


Who are Testers?

�Testers are the persons who:� Are usually not welcomed,� Do a boring repeated work,� Do a destructive process,� Bring bad news,� Describe bugs to authors,� Work under worst (time) pressure,� Need to take a different view,� Need a different mindset,


Who are Testers? (3)

�Testers have responsibilities to:� follow the test plans, script, etc. as

documented � report bugs objectively and factually, no abuse� check tests are correct before reporting bugs� check tests are correct before reporting bugs� remember it is the software, not the

programmer, that you are testing� assess risk objectively� prioritise what you report � communicate the truth


Who are Testers? (2)

�Testers have the rights to:� be regarded as a professional,� request specifications and test plans,� access to accurate progress and changes,� be trained in working domain,� find bugs,� report bugs seriously (non-reproducible),� make predictions about future bug levels,


Questions

The important thing is not to stopquestioning. Curiosity has its ownreason for existing.

(Albert Einstein)


Documents

Software Testing 1