Software Testing

Recap

• Testing methods / Types– Black Box testing– White Box testing– Incremental / Thread testing

• Testing levels Vs testing methods

• Testing Strategy

Contents

• Debugging Process• Test cases• What is a test case?• Writing better test cases• Examples

Debugging Process

• Debugging occurs as a consequence of successful testing• It is still very much an art rather than a science• Good debugging ability may be an innate human trait• Large variances in debugging ability exist• The debugging process begins with the execution of a test case• Results are assessed and the difference between expected and

actual performance is encountered– This difference is a symptom of an underlying cause that lies hidden– The debugging process attempts to match symptom with cause, thereby

leading to error correction

Why is Debugging so Difficult?

• The symptom and the cause may be geographically remote• The symptom may disappear (temporarily) when another error is

corrected• The symptom may actually be caused by non-errors (e.g., round-off

accuracies)• The symptom may be caused by human error that is not easily

traced

(continued on next slide)

Why is Debugging so Difficult?(continued)

• The symptom may be a result of timing problems, rather than processing problems

• It may be difficult to accurately reproduce input conditions, such as asynchronous real-time information

• The symptom may be intermittent such as in embedded systems involving both hardware and software

• The symptom may be due to causes that are distributed across a number of tasks running on different processes

Debugging Strategies

• Objective of debugging is to find and correct the cause of a software error

• Bugs are found by a combination of systematic evaluation, intuition, and luck

• Debugging methods and tools are not a substitute for careful evaluation based on a complete design model and clear source code

• There are three main debugging strategies– Brute force– Backtracking– Cause elimination

Strategy #1: Brute Force

• Most commonly used and least efficient method• Used when all else fails• Involves the use of memory dumps, run-time traces, and output

statements• Leads many times to wasted effort and time

Strategy #2: Backtracking

• Can be used successfully in small programs• The method starts at the location where a symptom has been

uncovered• The source code is then traced backward (manually) until the

location of the cause is found• In large programs, the number of potential backward paths may

become unmanageably large

Strategy #3: Cause Elimination

• Involves the use of induction or deduction and introduces the concept of binary partitioning– Induction (specific to general): Prove that a specific starting value is

true; then prove the general case is true– Deduction (general to specific): Show that a specific conclusion follows

from a set of general premises• Data related to the error occurrence are organized to isolate

potential causes• A cause hypothesis is devised, and the aforementioned data are

used to prove or disprove the hypothesis• Alternatively, a list of all possible causes is developed, and tests are

conducted to eliminate each cause• If initial tests indicate that a particular cause hypothesis shows

promise, data are refined in an attempt to isolate the bug

Test Cases

Test case

• A test case contains:– A sequence of Steps describing actions to be performed, – Test data to be used– An expected response for each action performed.

• Test Cases are written based on Business and Functional/Technical requirements, use cases and Technical design documents. 

• There can be 1:1 or 1:N or N:1 or N:N relationship between requirements and Test cases

• The level of details specified in test cases will vary depending on Organizations, Projects and also on the Test Case Template used OR on the Test Management tool being used in the project. 

Test case (cont…)

• Construction of Test Cases also helps in:– Finding issues or gaps in the requirements – Technical design itself.

• As Test Case construction activity would make tester to think through different possible Positive and Negative scenarios.

• It is test cases against which tester will verify the application is working as expected.

• Number of test cases to be created depends on the size, complexity and type of testing being performed

Test case examples• Test case for ATM

– TC 1 :- succesful card insertion.– TC 2 :- unsuccessful operation due to wrong angle card insertion.– TC 3:- unsuccesssful operation due to invalid account card.– TC 4:- successful entry of pin number.– TC 5:- unsuccessful operation due to wrong pin number entered 3 times.– TC 6:- successful selection of language.– TC 7:- successful selection of account type.– TC 8:- unsuccessful operation due to wrong account type selected w/r to that inserted card.– TC 9:- successful selection of withdrawl option.– TC 10:- successful selection of amount.– TC 11:- unsuccessful operation due to wrong denominations.– TC 12:- successful withdrawl operation.– TC 13:- unsuccessful withdrawl operation due to amount greater than possible balance.– TC 14:- unsucceful due to lack of amount in ATM.– TC 15:- un due to amount greater than the day limit.– TC 16:- un due to server down.– TC 17:- un due to click cancel after insert card.– TC 18:- un due to click cancel after insert card and pin no.– TC 19:- un due to click cancel after language selection,account type selection,withdrawl

selection, enter amount

• Test cases for a web page– Testing without entering any username and password– Test it only with Username– Test it only with password.– User name with wrong password– Password with wrong user name– Right username and right password– Cancel, after entering username and password.– Enter long username and password that exceeds the set limit of

characters.– Try copy/paste in the password text box.– After successful sign-out, try “Back” option from your browser.

Check whether it gets you to the “signed-in” page.

Some terminologies…

• Documented Test Cases are referred to as Test Scripts.

• Related test cases or test scripts are collectively referred to as Test Suite. 

Writing better test cases1. Requirement ID(s) being covered in the Test Case.2. Test Condition(s) and Expected Result(s) being exercised in the Test Case.3. Initial setup required for executing the test script. This could be

environment or data or configuration setup to be done before running the test case.

4. Post execution activities. For e.g.:- Delete the application user ‘WebAdmin’ after test execution is completed.

5. Priority (High, Medium and Low) of the Test Case. Priority will help the tester to decide which test case(s) have to be run earlier than others.

6. Complexity of the Test Case. It will help to identify and filter Test Cases based on complexity. This would help in assigning test cases to testers, before test execution.

7. Approximate time required for executing the test case. This entry is required from Project management perspective to track the productivity and also to ensure we can still meet the test execution deadlines.

8. Test Steps. This contains instruction on what actions to perform and what test data to use.

Writing better test cases (cont…)

9. Expected results. Each Test Step will have a corresponding Expected result field that would specify the expected response.

10.Actual result. Each Test Step will have a corresponding Actual result field. Tester would enter the details on the response he saw after executing the test step.

11.Test Step result. Typically this field would contain values Not Applicable, No Run, Passed, Failed or in progress etc

12.Test Case Version number13.Test case creation timestamp.14.Revision history. When and who wrote or modified the test case etc.15.Test Case status (Draft, completed, reviewed, Not Valid etc.)16.Test Case execution timestamps.17.Associated Defects. This field will help to identify what are the existing

defect(s) that are associated with the test case.18.Project Name19.Application Name

How to write test cases?

• Requirements

• Use cases

• Design documents

Understanding requirements• First understand Business requirements get the big picture of what

is being achieved from business perspective. • Read and understand how Business requirements are exploded into

functional requirements. • Go through Technical specification and UI design document to gain

understanding of how the business requirements get translated into application functionality.

• Once you do this exercise obviously you will get at least few questions or you may even discover few gaps in the requirements.

• Reach out to Business Analyst and Business User (as required) and get your questions answered. Now you are ready to go ahead with your Test Scenario identification.

• Identify Test Scenarios, Test Conditions and Expected results. Once you have sign-off on Test Scenarios, Test Conditions and Expected results, start creating Test Cases.

Design documents

• Also consider verifying the technical aspects mentioned in the Technical and look and feel aspects mentioned in UI design documents.

• Some of the technical aspects like:– Field Length– Validation– Data structures will be documented in-detail in

Technical specification.

• So, during test case construction, one has to incorporate additional validations as well.

How to write test cases?

• Get the test cases reviewed• Detail of each test case• Write test cases in plain English• Requirement traceability matrix (RTM)• Provide test data within the test cases• Test case version control• Prioritize test cases• Pre-requisites and clean up sections

unique-test-case-id: Test Case Title

Purpose Short sentence or two about the aspect of the system is being tested. If this gets too long, break the test case up or put more information into the feature descriptions.

Prereq Assumptions that must be met before the test case can be run. E.g., "logged in", "guest login allowed", "user testuser exists".

Test Data List of variables and their possible values used in the test case. You can list specific values or describe value ranges. The test case should be performed once for each combination of values. These values are written in set notation, one per line. E.g.: loginID = {Valid loginID, invalid loginID, valid email, invalid email, empty}password = {valid, invalid, empty}

Steps Steps to carry out the test. See step formating rules below. 1.visit LoginPage 2.enter userID 3.enter password 4.click login 5.see the terms of use page 6.click agree radio button at page bottom 7.click submit button 8.see PersonalPage 9.verify that welcome message is correct username

Notes and Questions:

NOTE QUESTION

Test case review!

• Get your Test Cases reviewed by Business Analyst and Business Users.

• It is always good to get inputs from Subject Matter Experts

• It is better to get review comments from Business Analysts and Business Users before testing starts rather than they pointing out at the deficiencies of your test cases during acceptance testing or Post implementation.

How detailed should be a test case?

• Test Cases steps should be as detailed as possible and should not be written at high level.

• Writing detailed test steps is very important considering the fact that the same person who wrote the test cases may not always execute the same test cases.

• If the test cases are not very detailed then the person who executes the test case for the first time will not be able to validate the system thoroughly as he/she might have not gone through the requirements and will test only as per the test case steps.

Example of detail• Example of a Test Case that is written at high level.

– Step No: 1Step Description: Login to test application with valid user id/passwordExpected Result: Home page is displayed

– Step No: 2Step Description: Click “Logout” link on home pageExpected Result: Login page is displayed

• Example of a Test case written that is written in detail.– Step No: 1

Step Description: Open URL https://www.example.com/login.aspExpected Result: Login page is displayed and contain the below fieldsa) “User Name” text fieldb) “Password” text fieldc) “Submit” button

– Step No: 2Step Description: Once Login page is displayed. Enter valid userid/password.a) Enter “user1” in “User Name” text fieldb) Enter “abc123″ in “password” text fieldExpected Result: a) Verify “User Name” is populated with text “user1”b) Verify text entered in “password” field is masked and is not readable.

– Step No: 3Step Description: Click “Submit” buttonExpected Result: Verify application “Home” page is displayeda) Verify “Home” page displays “Welcome user1” message on top of left navigation Menu.b) Verify Left Navigation menu contains links “Directory”, “Submission”, “Latest Links”, “Approve Links”, “Logout”.

– Step No: 4Step Description: Click “Logout” link on the left menu.Expected Result: Verify user is successfully logged out of the application and application login page is displayed https://www.example.com/login.asp

Why do we write test cases?• Test case creation could have two broad goals:

1. Test Cases are supposed part of the deliverable to the customer. TC goal credibility in this case. Typically UAT (acceptance) level.

2. Test Cases are for team internal use only. Typically System level testing. Testing efficiency should be the goal in this case. The idea is to write test cases based on design while code is incomplete, so that we could test product quickly once the code is ready.

• In case of agile development goal number one is not applicablea) TC are used internally, but the goal is credibility, not efficiency. It also means that TC are

dramatically reworked during test execution

Summary