Evolving an Elective Software Testing Evolving an Elective Software Testing Course: Lessons LearnedCourse: Lessons Learned

Edward L. JonesFlorida A&M UniversityTallahassee, FL USA

3rd Workshop on Teaching Software Testing

Agenda

Course Overview

Student Background

Driving Principles

Overview of Assignments

Course Reflection

Improvements

Assignment Walkthroughs

Course Overview

DESCRIPTION: The purpose of this course is to build skills necessary to perform software testing at the function, class and application level. Students will be taught concepts of black-box (functional and boundary) and white-box (coverage-based) testing, and will apply these concepts to small programs and components (functions and classes). Students will also be taught evaluative techniques such as coverage and mutation testing (error seeding). This course introduces the software engineering discipline of software quality engineering and the legal and societal issues of software quality.

Programming Focus

AUDIENCE: Not software testers but software developers. What distinguishes the course approach is that is stresses the programming aspect of software testing. A goal is to enhance and expand students’ programming skills to support activities across the testing lifecycle: C++ programming and Unix shell script programming to automate aspects of software testing.

Students just needed a course to take ...

Conceptual Objectives

The student shall understand The software testing lifecycle The relationship between testing, V&V, SQA Theoretical/practical limits of software testing The SPRAE testing framework Concepts and techniques for black-/white-box

testing Test case design from behavioral model Design patterns for test automation Test coverage criteria Issues of software testing management

Performance Objectives

The student shall be able to: Use the Unix development environment

Write simple Unix shell scripts

Design functional and boundary test cases

Develop manual test scripts

Conduct tests and document results

Write test drivers to automate function, object and

application testing

Evaluate test session results; write problem reports

Learning/Evaluation Activities

80% practice / 20% concepts Lectures (no text) Laboratory assignments

Unix commands and tools

Testing tasks

Examinations 2 Online tests

Final (online)

Amnesty period (1 test / 2 labs)

Student Background

Reality: 20 students Not particularly interested in testing Low programming skill/experience

Ideal: An interest in software testing Strong programming skills Scientific method (observation, hypothesis

forming) Sophomore or junior standing Desire for internship in software testing

My Perspective on Teaching Testing

Testing is not just for testers! In ideal world, fewer testers required

Developers have tester’s skills/mentality Testing overlays development process

No silver bullet … just bricks Simple things provide leverage No one-size-fits-all

Be driven by a few sound principles

Driving Principles

Testing for Software Developers Duality of developer and tester

Few Basic Concepts Testing lifecycle

Philosophy / Attitudes (SPRAE)

Learn By Doing Different jobs across the lifecycle

A Testing Lifecycle

Analysis

Design

Implementation

Execution

Evaluation

Specification

Test Strategy/Plan

Test Script, Data, Driver

Defect DataProblem Reports

Test Results

Test Cases

Experience Objectives

Student gains experience at each lifecycle

stage

Student uses/enhances existing skills

Student applies different testing

competencies

Competencies distinguish novices from the

experienced

A Framework for Practicing Software Testing

• Specification the basis for testing

• Premeditation (forethought, techniques)

• Repeatability of test design, execution, and evaluation (equivalence v. replication)

• Accountability via testing artifacts

• Economy (efficacy) of human, time and computing resources

Key Test Practices

Practitioner -- performs defined test

Builder -- constructs test “machinery”

Designer -- designs test cases

Analyst -- sets test goals, strategy

Inspector -- verifies process/results

Environmentalist -- maintains test tools &

environment

Specialist -- performs test life cycle.

Test Products

Test Report (informal) of manual testing

Test Scripts for manual testing

Test Log (semi-formal)

Application Test Driver (Unix shell script)

Unit/Class Test Driver (C++ program)

Test Data Files

Test Results (automated)

Bug Fix Log (informal)

Specification Products

Narrative specification

Specification Diagrams

Specification Worksheet (pre/post conditions)

Decision Tables

Control Flow Graphs

Assignments Target Skills

Observation Skills Systematic exploration of software behavior

Specification Skills Describe expected or actual behavior

Programming Skills Coding for development of test machinery

Test Design Skills Derive test cases from specification using

technique Team Skills

Work with other testers

Course Reflection

Testing is programming intensive

Testing requires analytical skills and facility with

mathematical tools

Testing generates data management problem that

is amenable to automation

Testing gives students advantage in entry-level

positions

Students take this course too late

Failed Course Expectations

Students test at all levels No “in-the-large” application (e.g., web-based)

Students develop intuitive testing skills On largest project, concepts did not transfer 1 in 3 students show “knack” for testing

Impact of balance of concept and experience Poor performance on exams with problems like those

in labs Test case design skills low

Homework needed v. labs (programming) Mentoring (timely feedback) did not occur

Students left to own devices too much

Why These Outcomes?

Formalisms important, but difficult Provide the behavior model (e.g., decision table) Basis for systematic test case design, automation

Lack of textbook Students need concepts + lots of examples

Poor availability when students were working Students worked at last minute Not always around Automated grading lacked 1-1 feedback

Standards-rich/tool-poor environment a distraction Assigned work too simple??

Proposed Changes

Improve lecture notes and example bank Find and refine

Resources and workbook

Outside-in: testing in-the-large before in-the-small

Recitation/laboratory for discussion and feedback

Increase use of testing tools (no-cost)

Increase use of collection of code/applications

Examination testbank for practice, learning

Assignment Walkthroughs

(see paper)

Assignment Walkthroughs

Blind Testing

Test Documentation

Specification

Test Automation via Shell Scripts

Unit Test Automation (Driver)

White-Box Unit Testing

Class Testing

Blind Testing I

Objective: Explore behavior of software without

the benefit of a specification

Given: Executables + general description

Results: Students not systematic in exploration

or in generalizing observed behavior Hello output based on length of input

Add 1-digit modulus 10 adder, input exception

Pay pay calculation with upper bound pay amount

Blind Testing II

Programming Objective: Student writes program

that matches the observed behavior of Blind

Testing I

Test Objective: Observations on Blind Testing I

used as “test cases” for reverse-engineered

program.

Results: Students did not see the connection; Did not replicate the recorded behavior

Did not recognize (via testing) failure to replicate

SUPPLEMENTAL SLIDES

Student work

SCALING UPThe heart of the approach is to use a decision table as a thinking tool. The most critical task in this process is to identify all the stimuli and responses. When there are many logical combinations of stimuli, the decision table can become large, indicating that the unit is complex and hard to test.

IDENTIFYING BEHAVIORApproaches

• Work backwards» Identify each response» Identify conditions that provoke response» Identify separate stimuli

• Work forward» Identify stimuli» Identify how each stimulus influences what

unit does»Specify the response

IDENTIFYING STIMULI

• Arguments passed upon invocation• Interactive user inputs• Internal, secondary data

»global or class variables• External data (sources)

» file or database status variables» file or database data

• Exceptions

IT PAYS TO BE A GOOD STIMULUS DETECTIVE

• Failure to identify stimuli results in an incomplete, possibly misleading test case

• The search for stimuli exposes» interface assumptions -- a major source of

integration problems

» incomplete design of unit

» inadequate provision for exception handling

IDENTIFYING RESPONSES

• Arguments/Results passed back on exit• Interactive user outputs• Internal, secondary data

»updated global or class variables• External data (sinks)

»output file or database status variables»output file or database data

• Exceptions

IT PAYS TO BE A GOOD RESPONSE DETECTIVE

• Failure to identify responses results in » incomplete understanding of the software

under test

»shallow test cases

» incomplete expected results

» incomplete test "success" verification -- certain effects not checked

• To test, one must know all the effects

A SKETCHING TOOL Black-Box Schematic

Stimulus Type Response Type

SoftwareunderTest

Argument

Inputs

Globals

Database

Exception

Argument

Outputs

Globals

Database

Exception

BEFORE CONTINUTINGMuch of the discussion so far involves how to identify what software does. We have introduced thinking tools for systematically capturing our findings. These thought processes and tools can be used anywhere in the lifecycle, e.g., in software design!

One Stone for Two Birds!!

Specialist I - Competencies

Practitioner

Test Designer

Test Analyst

Test Inspector

Test Environmentalist

Test SPECIALIST

1

Test Builder 1

1

1

1

1

1

2

2

2

2

2

2

2

3

3

3

3

3

3

3

4

4

4

4

4

4

4

5

5

5

5

5

5

5

Test Practitioner ...

...

...

...

...

...

...

BOUNDARY TESTING DESIGN METHODOLOGY

• Specification

• Identify elementary boundary conditions

• Identify boundary points

• Generate boundary test cases

• Update test script (add boundary cases).

EXAMPLE: Pay Calculation(1) Specification

• Compute pay for employee, given the number of hours worked and the hourly pay rate. For hourly employees (rate < 30), compute overtime at 1.5 times hourly rate for hours in excess of 40. Salaried employees (rate >= 30) are paid for exactly 40 hours.

EXAMPLE B(2) Identify Behaviors

• Case 1: Hourly AND No overtime » (Rate < 30) & (Hours <= 40)» Expect Pay = Hours * Rate

• Case 2: Hourly AND Overtime » (Rate < 30) & (Hours > 40)» Expect Pay = 40*Rate+1.5*Rate*(Hours - 40)

• Case 3: Salaried (Rate >= 30) » Expect Pay = 40 * Rate

DECISION TABLE

Condition

c1: Rate < 30 | Y Y N N

c2: Hours <= 40 | Y N Y N

Action

a1: Pay = Straight time | X

a2: Pay = Overtime | X

a3: Pay = Professional | X X

Columns defineBehaviors

EXAMPLE B(3) Create Test Cases

• One test case per column of decision table

» Case 1: Hourly, No Overtime

» Case 2: Hourly, Overtime

» Case 3: Salaried, No Extra Hours

» Case 4: Salaried, Extra Hours

• Order the test cases by column

EXAMPLE B(4) Write Test Script

StepStimuli Expected Response

Hours Rate Pay =

1

2

30

50 10

10 300

550

3 30 40 1600

4 50 40 1600

Testing Modules -- Drivers

A test driver executes a unit with test case data and captures the results.

Driver

Test setData External

EffectsUnitArguments

ResultsTest Set Results

Implementing Test Drivers

Complexity Arguments/Results only Special set-up required to execute unit External effects capture/inquiry Oracle announcing "PASS"/"FAIL"

Major Benefits Automated, repeatable test script Documented evidence of testing Universal design pattern

Test Driver for Unit Pay

Driver D_payuses unit_environment E;{ declare Hrs, Rate, expected; testcase_no = 0; open tdi_file("tdi-pay.txt"); open trs_file("trs-pay.txt"); while (more data in tdi_file) { read(tdi_file, Hrs, Rate); read(tdi_file, expected);

testresult = pay(Hrs, Rate); write (trs_file, testcase_no++, Hrs, Rate, expected, testresult); }//while close tdi_file, trs_file; }//driver

Test Driver Files (Pay)

Test Data FileFile name: tdi-pay.txtFormat: (test cases only) rate hours expected-pay

File content:10 40 400 10 50 55010 0 0

------Note: No environment setup.

Test Results FileFile name: trs-pay.txtFormat: case# rate hours exp-pay act-pay

File content:1 10 40 400 4002 10 50 550 5003 10 0 0 0

------Note: Results file must be inspected for failures.

PassFailPass

Test Script!!

Testing Classes -- Drivers(Black-Box)

ClassTest

Driver Method Args/Results

Test setData

Test Set Results

Class

Class-state

Method(s)

Example -- Stack Class

class Stack{ public: Stack(); void push(int n); int pop(); int top(); bool empty(); private: int Size; int Top; int Values[100];};

Notes:

(1) Class state -- variables Size, Top and the first 'Size' values in array Values.(2) Methods push and pop modify class state; top and empty inquire about the state.(3) Stack does not require any test environment of its own.(4) Class state HIDDEN from test, i.e., black box.

Test Driver Files (Stack class)

Test Data File (tdi-stack.txt)File content:----- 1 8 1 7 3 7 2 7 2 8 4 true------Note: No test environment setup.Methods: 1-push, 2-pop, 3-top, 4-empty

Test Results File (trs-stack.txt)File content:-----1 1 8 2 1 73 3 7 84 2 7 85 2 8 76 4 1------Note: Results file must be inspected for pass/fails.

--- Top should be 7.

--- Push .

--- Pop, should be 8.--- Stack should be empty.

FailFailFailPass