(1)

Verification MethodsCS2110: SW Development Methods

•Reading: Study Section 1.2.5 in the MSD text •These slides supplement the textbook

– These touch on other important methods, but focus on execution-based testing

(2)

Some Definitions

• Verification: showing that something meets specifications– We verify “against” something– Important: if you don’t have requirements

specifications documented, then you can’t verify!– If behavior is not defined, then what counts as

incorrect behavior?• Verification can happen on any kind of software work-

product– designs, code, etc.

(3)

Verification Techniques

At least four approaches to verifying code quality:1. Execution-based Testing (running tests)2. Inspections

– People examine the work product by eye3. Static Analysis by a tool

– For C programs, an old tool called “lint”– For Java programs, some nice tools integrate

with Eclipse: CheckStyle and PMD4. Formal Correctness Proofs

– Use mathematics to demonstrate the desired property

(4)

Execution-based Testing

Some Basic Definitions:• Fault: What we think of as a bug

– Something in the code that’s wrong• Failure: Observed incorrect behavior of a system

resulting from a fault• Error: Mistake made by a developer

A possible definition of execution-based testing:– Process of inferring certain behavioral properties

of a product based, in part, on results of executing it in a known environment with selected inputs.

(5)

Famous and Important Quote

• Program testing can effectively show the presence of bugs but is hopeless for showing their absence.– Edsger Dijkstra

– Famous computer scientist– Turing Award winner– Wrote "Go To Statement

Considered Harmful“ (1968)– Shortest-path algorithm

(6)

Types of Testing

• Unit testing– An individual class or function is tested thoroughly– Done as part of coding process– Done by the programmer him/herself– Usually by a test harness (or driver) or using a

test-class• Integration testing

– Multiple parts of a system are combined and tested to see if they work together

• Acceptance testing– The customer is involved. (Alpha or beta testing)

(7)

Approaches to Unit Testing

• Two approaches– Test only based on inputs, outputs (i.e. the

specification) without looking at the code– Test based on the logic of the code (must study it)

• The 1st of these is called black-box testing• The 2nd of these is glass-box or white-box testing• Some principles of glass-box testing

– Make sure every line (or “case”) gets executed– Make sure loops get done zero times, >zero times– Etc.

(8)

Limitations of Glass-Box Testing

• Why not fully exercise our code? • Logic of a piece of code defines a set of possible

execution paths, “flow-paths,” through the function– Think of a flow-chart– The inputs control which path is taken

• A good set of test data might make sure every possible path is followed– This tests every possible behavior

• Problem: for even small programs with loops and conditionals, the total number of paths becomes too large too quickly– See MSD text, p. 24-25

(9)

Black-Box Testing

• This means testing based on inputs to the code-module without basing tests on code’s internals– You can’t see inside a black-box– Note: MSD’s definition on p. 26 isn’t the usual one

• Choose test cases for common problems– Boundary values, boundary conditions– Off-by-one errors– Incorrect inputs

• Important: You have to know in advance what the expected result of any test should be!

(10)

Lab on Monday/Tuesday

• The JUnit (v. 3) framework is a tool to help people generate and format unit tests in a systematic way.

• Stand-alone and alsobuilt into Eclipse

• Example JUnit test– Class contains all

tests– setUp() performs

preconditions– “test…” is an

individual test case– “assert” checks a

program condition

public class CoffeeMakerTest extends TestCase { private CoffeeMaker cm; private Recipe recipe1;

public void setUp() { cm = new CoffeeMaker();

recipe1 = new Recipe(); recipe1.setName("Coffee"); recipe1.setPrice(50); recipe1.setAmtCoffee(6); recipe1.setAmtMilk(1); }

public void testAddRecipe() { assertTrue(cm.addRecipe(recipe1)); }}

(11)

(12)

When we test a class’ method to see if its loop goes to size-1 and not size, is this

clearly an example of:

1. A black box test

2. A glass-box test

3. A unit test

4. Both 1 and 3

5. Both 2 and 3

(13)

Planning Your Tests

• How do we plan a “test case”?Define three parts first:– A purpose (why are we doing this particular test?)– An input– An expected result

• We execute the test to see if observed output matched the expected result– It’s really just like a scientific experiment

• When testing, have a plan (at least in your head)– Don’t test blindly or randomly

(14)

Planning Your Tests

• Poorly specified test cases…

Test ID Description Expected Results

Actual Results

1 Player 1 rolls dice and moves.

Player 1 moves on the board.

2 Player 2 rolls dice and moves.

Player 2 moves on the board.

(15)

Planning Your Tests

• Why are these test cases better than the previous ones?

Test ID Description Expected Results

Actual Results

1 Precondition: Game is in test mode, GameBoard is loaded, and game begins.

Number of Players: 2

Player 1 dice roll: 3

Player 1 moves to square Blue 3.

2 Precondition: TestID 1 completed successfully.

Player 2 dice roll: 3

Player 2 moves to square Blue 3.

(16)

Unit Testing: Boundary values

• Often programs misbehave because we make mistakes dealing with the “extremes”

• Examples:– The last item or first item in a collection– The start or end of a range of values

• Equivalent classes of test cases– Don’t have test cases or data that test the same

situation (e.g. the middle of a range)• Boundary or extreme conditions

– An empty list, an empty file– One item in a list or file

(17)

Unit Testing: Off-by-one errors

• Related to boundary conditions• Using the nth value instead of n-1• Using 0 instead of 1• Executing a loop body once when it the body should

not be executed at all– Wait – is this one a black-box testing method?

Why or why not?• Etc.

(18)

Unit Testing: Invalid Inputs

• Rely on pre-conditions defined for your functions– They specify what the agreement is about what

you are responsible for when its time to handle mis-use

• If a function is mis-used in this way– Return an error condition if the interface provides

a way to do this– Otherwise, halt program! Better than producing

unexpected results• Use assertions (more later!)

(19)

Example: Program with Student and Course classes

• Homework 1 this term– Define class Student

• methods include:

– Define class Course• methods include:

• Focus on defining:– Black-box test-cases for the method

boolean drop(Course c) in class Student• Let’s make a list of possible test-cases for this

method!

(20)

Example of Full Test-Case Description

• Test Case #1: Drop last course in list with >1 item– Purpose:  Drop Course c for Student s where that

course is the last course stored in the Student object's courses list.  This tests the boundary condition of last item in list for Student.drop(c).

– Input: Student s is enrolled in three courses, and the course to be dropped is the last of these stored in the list.

– Expected output:   The Student s is unchanged except that only two courses are stored for that student. Also, the list of students stored for that course is altered by having that student removed.

(21)

Other Test Cases for this Program

• There are a lot!• Boundary conditions?

– for Student.drop(): end of list, start of list– same for Course.drop()– Question: affected by whether you used ArrayList

instead of Java arrays?– Note: also test “typical” case, item in middle of list– Empty lists: Course.cancel(), Student.dropAll()

• Off-by-one errors?– Really included in boundary condition testing here

(22)

Other Test Cases for this Program

• Invalid inputs?– See the specifications!– Test return values

• For boolean, test both success and failure• Remember: make it fail!

• Test of equals() methods?– If you get this wrong, trouble!– Easy to test. (Easy to forget to.)

(23)

Good or Bad Test?

1. Good Test?

2. Poor Test?

Student s adds a course and the course appears in the schedule

(24)

Good or Bad Test?

1. Good Test?

2. Poor Test?

Student s currently is inthe system and currentlyhas no courses.Student s adds a course number 394 and it appearsat the end of the schedule.

(25)

Good or Bad Test?

1. Good Test?

2. Poor Test?

Student s has 4 courses,drops 3, adds 1. Check if thestudent is then deletedfrom the system.

(26)

Extra Slides

(27)

#2: Inspections: We Might See These Later

• Many different “styles” or approaches– Walk-throughs, Inspections

• Advantages:– You can inspect all of a product but usually it is

impossible to test all of a product.– Applicable to all work products (e.g. code,

designs, requirements documents, etc.)– Fairly easy: training, application, follow-up

• Disadvantages:– Staff time is expensive– Works best if follows a rigorous process