Software testing fundamentals

Software Testing and

Quality AssuranceSoftware Testing Fundamentals

Prof. Shailesh T. GahaneAssistant Professor

Dr. D. Y. Patil School of MCACharholi (Bk), Lohegaon, Pune – 412105

Mail: [email protected]

Prof. Shailesh T. Gahane, Dr. D Y Patil School of MCA, Pune


Software Testing Fundamentals

4.1 Definition & Objectives 4.2 Types of software bugs 4.3 Bug life cycle 4.4 Testing lifecycle 4.5 Test Plan 4.6 Test Cases – Definition, Test Case Designing 4.7 Case Studies on Test Plan & Test Case


What is Software TestingSoftware testing is a process used to identify the

correctness, completeness and quality of developed computer software.

It is the process of executing a program / application under positive and negative conditions by manual or automated means.

It checks for the :- Specification Functionality Performance

WHY SOFTWARE TESTING ?

Software Testing is important as it may cause mission failure, impact on operational performance and reliability if not done properly.

Effective software testing delivers quality software products satisfying user’s requirements, needs and expectations



TESTING OBJECTIVES

Uncover as many as errors (or bugs) as possible in a given timeline.

Demonstrate a given software product matching its requirement specifications.

Validate the quality of a software testing using the minimum cost and efforts.

Generate high quality test cases, perform effective tests, and issue correct and helpful problem reports.


Software Testing Scope

Software Testing Process

Software Testing Management

ConfigurationManagement

Software Problem Management

Software Testing Methods

Software Test Criteria

Software TestingTools

Software Test Models

Software Testing Strategies


Bugs are more insidious (deceiving but harmful) than ever we expect them to be.

An unexpected test result may lead us to change our notion of what a bug is and our model of bugs.

Some optimistic notions that many programmers or testers have about bugs are usually unable to test effectively and unable to justify the dirty tests most programs need.

even if we correctly discover all cases for placing words on the Scrabble board, it is very likely that we'll have some bugs when we code it bugs are inevitable in any complex software system a bug can be very visible or can hide in your code until a much later

date

INTRODUCTION AND DEFINITION BUGS :


EXAMPLES OF RELATED BUGS AND ISSUES

“A snippet that enunciates a horizontal scaling problem” (Skinner) Problem: “Anyone who placed an order and subsequently went back to update the order could frequently not find that order. Also, when a customer changed an address on the Web site and later returned to place an order, the order was often associated with the old address.

Cause: “A technician explained that requests made by the Web site were randomly distributed across the six database servers. When a visitor returned to the site to view an order, there was only a one in six chance that the visitor's request would be routed to the correct database server. The basic problem was that when a single database server was queried about an order, it needed (but did not have) the information held on the other database servers. Because of this, the database was not a good candidate for horizontal scaling.” (Skinner


CAUSES OF BUGS Incomplete or Erroneous Specification(IES) : The root causes of s/w bugs is the incorrect requirement specification or

constantly changing requirements. Misinterpretation of customer communication(MCC) :\ Requirements are not communicated well to the entire development team. The next main cause bugs is the design. International Deviation From Specification(IDS). Violation of programming Standards(VPS). Error in data representation(EDR). Inconsistent Module Interface(IMI). Error in design Logic(EDL) Incomplete or Erroneous Testing(IET). Inaccurate or Incomplete Documentation(IID). Error In programming Language Translation of Design(PLT).


TYPES OF BUGS Compile Time : Syntax, spelling static type mismatch. Such bugs are

caught with compiler. Design :Flawed algorithm. Ex. incorrect outputs Program Logic : (If/else loop termination, select case, etc.)

Ex. incorrect outputs Memory Nonsense : Null pointers, array bounds, bad types, leaks.

Ex. Runtime exceptions. Interface errors between modules, threads, program with shared

resources: Ex. Runtime exceptions. Off-Nominal Conditions: failure of some part of software of

underline machinery(network etc.) Ex. Incomplete functionality.

Deadlock's :Multiple processes fighting for a resources.

Ex. Freeze ups, never ending processes.


BUG LIFE CYCLE

• Do it until solved • New • Open • In-Fix • Fix-Complete • In-Retest • Retest-Complete • Closed • Retest-Complete • Open •


YOUR PROGRAM

Should produce the desired results for all legal inputs Should give reasonable error messages for illegal inputs Need not worry about misbehaving hardware Need not worry about misbehaving system software Is allowed to terminate after finding an error

3, 4, and 5 are true for beginner’s code; often, we have to worry about those in real software


SOURCE OF ERRORS Poor specification

“What’s this supposed to do?” Incomplete programs

“but I’ll not get around to doing that until tomorrow” Unexpected arguments

“but sqrt() isn’t supposed to be called with -1 as its argument”

Unexpected input “but the user was supposed to input an integer”

Code that simply doesn’t do what it was supposed to do “so fix it!”


Compilation errors Syntax error (example: missing a semi-colon). Semantic error. (For example, applying modulus % on floating-point value for certain programming languages. In Java ,is it fine? Yes!) Easiest type of errors to fix.

Runtime errors Occur at runtime. Java’s exception mechanism can catch such errors.

Logic errors Program runs but produces incorrect result. Hard to characterize, hence hardest to fix.

Programming errors are also known as bugs Origin: a moth in the Mark I computer.

TYPES OF ERRORS

Testing

Yes

Error?

Debug


Path Testing Paths: different routes that your program can take

Design test data to check all paths Example

if (x != 3) { y = 5;}else { z = z - x;}

if (z > 1) { z = z / x;}else { z = 0;}

if (x != 3)

y = 5 z = z - x

if (z > 1)

z = z / x z = 0

A

B

C

D

E

F

G

H

<x=0, z=1> Paths A, B, G, H.

<x=3, z=3> Paths E, F, C, D.


17

FaultsWhat is the difference between a fault and

an error? What are some kinds of faults?

Error : incorrect software behavior. example: message box text said "Welcome null.“

Fault : mechanical or algorithmic cause of error example: account name field is not set properly. a fault is not an error, but it can lead to them need requirements to specify desired behavior, and

need to see system deviate from that behavior, to have a failure


WHAT IS BUG? A computer bug is an error, flaw, mistake,

failure, or fault in a computer program that prevents it from working correctly or produces an incorrect result. Bugs arise from mistakes and errors, made by people, in either a program’s source code or its design.

A fault in a program, which causes the program to perform in an unintended or unanticipated manner.

The general definition of bug is: “failure to conform to specifications”

If you want to detect and resolve the defect in early development stage, defect tracking and software development phases should start simultaneously.


LIFE CYCLE OF A BUG? When tester logs any new bug the mandatory

fields are:Build version, Submit On, Product, Module, Severity, Synopsis and Description to Reproduce

If you have authority to add bug Status, Priority and ‘Assigned to’ fields them you can specify these fields. Otherwise Test manager will set status, Bug priority and assign the bug to respective module owner.



BUG STATUS DESCRIPTION

1) New: When QA files new bug.

2) Deferred: If the bug is not related to current build or can not be fixed in this release or bug is not important to fix immediately then the project manager can set the bug status as deferred.

3) Assigned: ‘Assigned to’ field is set by project lead or manager and assigns bug to developer.


4) Resolved/Fixed: When developer makes necessary code changes and verifies the changes then he/she can make bug status as ‘Fixed’ and the bug is passed to testing team.5) Could not reproduce: If developer is not able to reproduce the bug by the steps given in bug report by QA then developer can mark the bug as ‘CNR’. QA needs action to check if bug is reproduced and can assign to developer with detailed reproducing steps.6) Need more information: If developer is not clear about the bug reproduce steps provided by QA to reproduce the bug, then he/she can mark it as “Need more information’. In this case QA needs to add detailed reproducing steps and assign bug back to dev for fix.


7) Reopen: If QA is not satisfy with the fix and if bug is still reproducible even after fix then QA can mark it as ‘Reopen’ so that developer can take appropriate action.8 ) Closed: If bug is verified by the QA team and if the fix is ok and problem is solved then QA can mark bug as ‘Closed’.9) Rejected/Invalid: Some times developer or team lead can mark the bug as Rejected or invalid if the system is working according to specifications and bug is just due to some misinterpretation.


HOW BUGS GET INTO SOFTWARE?

In software development projects, a "mistake" or "fault" can be introduced at any stage during development. Bugs are a consequence of the nature of human factors in the programming task. They arise from oversights or mutual misunderstandings made by a software team during specification, design, coding, data entry and documentation.


WHAT IS SOFTWARE TESTING?

Software Testing is a process of evaluating a system by manual or automatic means and verify that it satisfies specified requirements or identify differences between expected and actual results.


STLC is simply a testing phase in the SDLC development.

Validation and Authentication is tried and tested in this phase.

The only limitation of this cycle is that it is limited to respective individual phase and is carried out by a group of skilled testers.

What is STLC?



Phase Activity Deliverables

Requirements/Design Review

You review the software requirements/design

Review Defect Reports

Test Planning

Once you have gathered a general idea of what needs to be tested, you ‘plan’ for the tests.

Test Plan Test Estimation Test Schedule

Test Designing

You design/detail your tests on the basis of detailed requirements/design of the software (sometimes, on the basis of your imagination).

Test Cases/ Test Scripts/Test Data

Requirements Traceability Matrix


Phase Activity Deliverables

Test Environment Setup

You setup the test environment (server/client/network, etc) with the goal of replicating the end-users’ environment.

Test Environment

Test Execution

You execute your Test Cases/Scripts in the Test Environment to see whether they pass.

Test Results (Incremental)

Defect Reports

Test Reporting

You prepare various reports for various stakeholders.

Test Results (Final) Test/Defect Metrics Test Closure Report Who Worked Till

Late & on Weekends Report


Requirement Analysis In this phase testers analyze the customer

requirements and work with developers during the design phase to see which requirements are testable and how they are going to test those requirements.

It is very important to start testing activities from the requirements phase itself because the cost of fixing defect is very less if it is found in requirements phase rather than in future phases.

Test PlanningThis phase is also called Test Strategy phase.In this phase all the planning about testing is

done like what needs to be tested, how the testing will be done, test strategy to be followed, what will be the test environment, what test methodologies will be followed, hardware and software availability, resources, risks etc.


Test Designing

In this phase various black-box and white-box test design techniques are used to design the test cases for testing.

Testers start writing test cases by following those design techniques, if automation testing needs to be done then automation scripts also needs to written in this phase.

Test Environment SetupTest environment decides the software and

hardware conditions under which a work product is tested.

Test environment set-up is one of the critical aspects of testing process and can be done in parallel with Test Case Development Stage.


Test Execution During this phase test team will carry out the

testing based on the test plans and the test cases prepared.

Bugs will be reported back to the development team for correction and retesting will be performed.

Test Reporting Testing team will meet , discuss and analyze

testing artifacts to identify strategies that have to be implemented in future.

After tester assures that defects have been fixed and no more critical defects remain in software the build is given for final testing.



DEFINITION:A test plan documents the strategy that will be used to verify and ensure that a product or system meets its design specifications and other requirements. A test plan is usually prepared by or with significant input from test engineers

A test plan is a document detailing a systematic approach to testing a system such as a machine or software. The plan typically contains a detailed understanding of the eventual workflow.

Test Plan


http://en.wikipedia.org/wiki/Machine

http://en.wikipedia.org/wiki/Software

http://en.wikipedia.org/wiki/Workflow

1: We create a test plan because preparing it helps us to think the efforts needed to validate the acceptability of a software product. 2: We create a test plan to understand the why and how of product validation. 3: We create a test in regulated environments , because we need to have a written test plan. 5: We create a test plan because we want a document that describes the objectives, scope, purpose and focus of the software testing effort. 6: We create a test plan because it includes test cases, conditions, the test environment , pass/fail criteria, and risk assessment.

PURPOSE OF TEST PLAN


Advantages of Test Plan

Serves as a guide to testing throughout the development.We only need to define test points during the testing phase.Serves as a valuable record of what testing was done.The entire test plan can be reused if regression testing is

done later on.

In software testing, a test plan gives detailed testing information regarding an upcoming testing effort, including

1. Scope of testing2. Schedule3. Test Deliverables4. Risks and Contingencies Prof. Shailesh T. Gahane, Dr. D Y Patil School of MCA, Pune

TEST PLAN FEATURES

The test plan keeps track of possible tests that will be run on the system after coding.

The test plan is a document that develops as the project is being developed.

The initial test plan is abstract and the final test plan is concrete.

The most important test cases come from the requirements of the system.

When the system is in the design stage, the initial tests can be refined a little.

During the detailed design or coding phase, exact test cases start to materialize.

After coding, the test points are all identified and the entire test plan is exercised on the software.

TEST PLAN TYPES

One can have the following types of test plans:

Master Test Plan: A single high-level test plan for a project/product that unifies all other test plans.

Testing Level Specific Test Plans: Plans for each level of testing.

1. Unit Test Plan2. Integration Test Plan3. System Test Plan4. Acceptance Test Plan

Testing Type Specific Test Plans: Plans for Performance Test Plan and Security Test Plan.


TEST PLAN GUIDELINES

• Make the plan concise. • Avoid redundancy .• Make use of lists and tables wherever possible. • Avoid lengthy paragraphs.• Have the test plan reviewed a number of times prior

to base lining it or sending it for approval.• Update the plan as and when necessary.


Write An Introduction - An introduction includes a general description and schedule of a test, as well as any related documents.

Write A Section On Required Resources - This section describes all of the resources needed to complete the testing, including hardware, software, testing tools, and staff.

Write A Section On What You Are Going To Test - List what new aspects you will be testing and what old aspects you will be re-testing.

Write A Section On What You Will Not Be Testing.

7 Steps for writing a test plan


Write a section on the results of your project -Outline all the goals that you hope to achieve during the testing process. Detail the parameters for which success and failure can be measured.

Write a section on the results of your project -Outline all the goals that you hope to achieve during the testing process. Detail the parameters for which success and failure can be measured.

Write a section on the results of your project - Outline all the goals that you hope to achieve during the testing process. Detail the parameters for which success and failure can be measured.




Mobile App Development Test Plan


Tree Diagram For Testing


DEFINATION

A test case is a set of conditions or variables under which a tester will determine whether a system under test satisfies requirements or works correctly.The process of developing test cases can also help find problems in the requirements or design of an application

TEST CASES


TEST PLANThe goal of test planning is to establish the list of tasks

which, if performed, will identify all of the requirements that have not been met in the software. The main work product is the test plan. The test plan documents the overall approach to the test.

In many ways, the test plan serves as a summary of the test activities that will be performed.

It shows how the tests will be organized, and outlines all of the testers’ needs which must be met in order to properly carry out the test.

The test plan should be inspected by members of the engineering team and senior managers.


DEFINATION:“Test case is a description of a specific interaction that a tester will

have in order to test a single behavior of the software.” Test cases are very similar to use cases, in that they are step-by-step

narratives which define a specific interaction between the user and the software. A typical test case format is shown in a table, and it includes:

• A unique name and number• A requirement which this test case is exercising• Preconditions which describe the state of the software before the test

case (which is often a previous test case that must always be run before the current test case)

• Steps that describe the specific steps which make up the interaction• Expected Results which describe the expected state of the software

after the test case is executed Test cases must be repeatable.

Good test cases are data-specific, and describe each interaction necessary to repeat the test exactly.

TEST CASES

FORMAT OF TEST CASES

Test Case ID: Fun_10 Test Designed by: <Name>

Test Priority (Low/Medium/High): Med

Test Designed date: <Date>

Module Name: Google login screen

Test Executed by: <Name>

Test Title: Verify login with valid username and password

Test Execution date: <Date>

Description: Test the Google login page

Pre-conditions: User has valid username and passwordDependencies:


Test case ID: Unique ID for each test case. Follow some convention to indicate types of test. E.g. ‘TC_UI_1′ indicating ‘user interface test case #1′.Test priority (Low/Medium/High): This is useful while test execution. Test priority for business rules and functional test cases can be medium or higher whereas minor user interface cases can be low priority. Test priority should be set by reviewer.Module Name – Mention name of main module or sub module.Test Designed By: Name of testerTest Designed Date: Date when wroteTest Executed By: Name of tester who executed this test. To be filled after test execution.Test Execution Date: Date when test executed.Test Title/Name: Test case title. E.g. verify login page with valid username and password.Test Summary/Description: Describe test objective in brief.Pre-condition: Any prerequisite that must be fulfilled before execution of this test case. List all pre-conditions in order to successfully execute this test case.Dependencies: Mention any dependencies on other test cases or test requirement.

DESCRIPTION


Test case example for the Login page:


Case Study on Test Case and Test Plan

Test Plan

A test plan is a document detailing a systematic approach to testing a system such as a machine or software. The plan typically contains a detailed understanding of the eventual workflow

http://en.wikipedia.org/wiki/Machine

http://en.wikipedia.org/wiki/Software

http://en.wikipedia.org/wiki/Workflow


Depending on the product and the responsibility of the organization to which the test plan applies, a test plan may include a strategy for one or more of the following:• Design Verification or Compliance test - to be performed during the

development or approval stages of the product, typically on a small sample of units.

• Manufacturing or Production test - to be performed during preparation or assembly of the product in an ongoing manner for purposes of performance verification and quality control.

• Acceptance or Commissioning test - to be performed at the time of delivery or installation of the product.

• Service and Repair test - to be performed as required over the service life of the product.

• Regression test - to be performed on an existing operational product, to verify that existing functionality didn't get broken when other aspects of the environment are changed (e.g., upgrading the platform on which an existing application runs).



Test Case Definition

A test case is a set of conditions or variables under which a tester will determine whether a system under test satisfies requirements or works correctly.

The process of developing test cases can also help find problems in the requirements or design of an application.


Test Case Template

A test case can have the following elements. Note, however, that normally a test management tool is used by companies and the format is determined by the tool used.


Test Suite ID The ID of the test suite to which this test case belongs.

Test Case ID The ID of the test case.

Test Case Summary The summary / objective of the test case.

Related Requirement The ID of the requirement this test case relates/traces to.

Prerequisites Any prerequisites or preconditions that must be fulfilled prior to executing the test.

Test Procedure Step-by-step procedure to execute the test.

Test Data The test data, or links to the test data, that are to be used while conducting the test.

Expected Result The expected result of the test.

Actual Result The actual result of the test; to be filled after executing the test.

Status Pass or Fail. Other statuses can be ‘Not Executed’ if testing is not performed and ‘Blocked’ if testing is blocked.

Remarks Any comments on the test case or test execution.

Created By The name of the author of the test case.

Date of Creation The date of creation of the test case.

Executed By The name of the person who executed the test.

Date of Execution The date of execution of the test.

Test Environment The environment (Hardware/Software/Network) in which the test was executed.


EXAMPLE OF TEST CASE

Test Suite ID TS001 Test Case ID TC001

Test Case Summary To verify that clicking the Generate Coin button

generates coins. Related Requirement RS001

Prerequisites 1. User is authorized.

2. Coin balance is available.

Test Procedure

1. Select the coin denomination in the Denomination field.

2. Enter the number of coins in the Quantity field.

3. Click Generate Coin.

Test Data 1. Denominations: 0.05, 0.10, 0.25, 0.50, 1, 2,

5 2. Quantities: 0, 1, 5, 10, 20

Expected Result

1. Coin of the specified denomination should be produced if the specified Quantity is

valid (1, 5) 2. A message ‘Please enter a valid quantity

between 1 and 10′ should be displayed if the specified quantity is invalid.

Actual Result

1. If the specified quantity is valid, the result is as expected.

2. If the specified quantity is invalid, nothing happens; the expected message is not

displayed Status Fail

Remarks This is a sample test case. Created By John Doe

Date of Creation 01/14/2020 Executed By Jane Roe

Date of Execution 02/16/2020

Test Environment OS: Windows Y

Browser: Chrome N


CASE STUDY DEFINITION

Documented study of a specific real-life situation or imagined

scenario, used as a training tool in business schools and firms.

Students or trainees are required to analyze the prescribed cases

and present their interpretations or solutions, supported by the

line of reasoning employed and assumptions made.

http://www.businessdictionary.com/definition/study.html

http://www.businessdictionary.com/definition/scenario.html

http://www.businessdictionary.com/definition/training.html

http://www.businessdictionary.com/definition/tool.html

http://www.businessdictionary.com/definition/business-school.html

http://www.businessdictionary.com/definition/required.html

http://www.businessdictionary.com/definition/analyze.html

http://www.businessdictionary.com/definition/case.html

http://www.businessdictionary.com/definition/solution.html

http://www.businessdictionary.com/definition/employment.html

http://www.businessdictionary.com/definition/assumptions.html


TYPES OF CASE STUDIES Explanatory: Used to do causal investigations.

Exploratory: A case study that is sometimes used as a prelude to further, more in-depth research. This allows researchers to gather more information before developing their research questions and hypotheses.

Descriptive: Involves starting with a descriptive theory. The subjects are then observed and the information gathered is compared to the pre-existing theory.

Intrinsic: A type of case study in which the researcher has a personal interest in the case.

Collective: Involves studying a group of individuals.

Instrumental: Occurs when the individual or group allows researchers to understand more than what is initially obvious to observers.


What Does a Software Case Study Look Like?The structure of the most common type of commercial case study.Introduction or description of situation – This is where the customer – is introduced, along with their general status or situation. This sets the stage for the type of stakeholder being discussed.Description of problem and stakeholder’s goal – Here, we describe the problem that is being faced in this case study. What needs to be solved, fixed, reduced, increased, eliminated, or happen, in general?The proposal – our vendor’s product or service is introduced as a possible solution to the problem. The reasons for the vendor’s product being a “fix” are described.The solution – we now tell a very brief “story within a story” describing how the vendor assisted the customer. Even if the product is technology-based, we want a “big story” – the general actions that were taken to solve the problem. Please note that we’re not trying to gloss over details – we just want to keep focus on the expertise and the quality of our vendor.The conclusion – our story concludes (hopefully) with a bang. We can now make concluding statements that describe the benefits that the customer enjoys, and the ongoing success that the customer will reap as a consequence of the services employed during this case.


Case Study Example: The System Security Tool Introduction: Initech LLC is a 90 million dollar per year privately owned vendor

of enterprise banking systems. Problem Statement: Initech was experiencing an abundance of employee-injected

viruses and worms on its production servers on a regular basis. The Proposal: An expert consultant was brought in to recommend a solution. He

proposed that a product called Rid-It Pro, made by BugBGone, Inc. was an effective solution for the removal of viruses and malware and has extensive monitors to detect employee abuse.

The Solution: BugBGone’s flagship product “Rid-It Pro” was deployed on Initech’s servers with a minimum of downtime and customer expense. Rid-It Pro effectively identified one employee in possession of key system passcodes who was responsible for the injection of worms. The viruses were removed immediately upon discovery of the cause.

The Conclusion: The use of Rid-It Pro prevented several hundred thousand dollars of further losses. Rid-It Pro’s forensic capabilities lead to the recovery of virtually all monies stolen. The evidence gathered by Rid-It Pro was successfully used in a termination action and pending lawsuit.


Explain Testing Life Cycle in detail. What are the different steps involved in Software Testing Life Cycle? Explain various steps involved in Bug Life Cycle? Prepare detail test plan for web based Railway reservation system having

following features:a) Train information search with respect to train number, route.b) Reservation availability and fair information.c) Online booking and cancellation of tickets.

System must ensure the security with reference to online transactions. Write a detailed test plan for University result declaration system, which

provides the facility to view students result (sem/year) wise, system should have facility to apply for rechecking/re-evaluation with payment. Your plan should include the desired test documents, test cases and test strategies.

Write a test plan with scope of testing, risks, strategy, schedule, pass/fail criteria for a online shopping system.

Write a test plan with scope of testing risks and contingencies, strategy, schedule, staffing and training, deliverables, item pass/fail criteria for Online Library System.

Prof. Shailesh T. GahaneAssistant Professor

Dr. D. Y. Patil School of MCACharholi (Bk), Lohegaon, Pune – 412105

Mail: [email protected]


Education

Software testing fundamentals