CSC 395 –Software Engineering

Lecture 9:

Testing -or-

How I Stopped Worrying and Learned to Love the Bug

Today’s Lecture

Discusses testing in early workflows Starts to answer important questions:

When and how should we do testing? Who should be responsible for performing tests? How do we know something is correct? What faith should we place in testing?

What Do We Test For

Obviously, need to test for correctness What is correct? Who decides?

Important to consider other issues, too Utility - is it easy to use, cost effective, & useful? Reliability - how often will it not work? Robustness - do we rely on specific environment? Performance - is it within space & time

constraints?

How Do We Do This Testing?

There are two basic types of testing Execution-based testing Non-execution-based testing

Testing performs two important actions Verification - Was workflow completed correctly? Validation - Does product satisfies requirements?

Terminology Warning: “Verify” also used for non-execution-based testing

Testing Team

Very useful to have independent testing team Experts on process of testing Insure developers & analysts deliver high-quality Be involved over entire life-cycle

Critical to have managerial independence Bugs, faults, and errors are part of development Testing team must be free to make decisions Arguments with your Boss are hard to win Final decisions should not be made by developer

or tester

Walkthroughs

Walkthrough team ideally has 4 - 6 people Member of team performing current workflow Member of team performing next workflow Member of SQA team Often includes a client Should NOT contain manager, partner, or boss

Walkthroughs must be thoroughly prepared Provide items not understood Provide items that appear to be incorrect

Walkthrough Meeting

Chaired by SQA member Only interest is in finding faults Practiced in running these meetings

Meeting detects any possible faults Should not last longer than 2 hours

“Committee is a group that keeps minutes and loses hours” -- Milton Berle

Some flagged items may be correct Cost of meeting is high, but good if faults found

“A camel is a horse designed by committee” -- Unknown

Running Walkthrough

Document-driven & not participant-driven Focus on determining if documents are correct Relies on knowledge brought by participants Eases transfer of responsibilities between teams

Faults often found describing documents Often discover problems by talking things through Many times bugs discovered

Inspections

Follows five formal steps1. Current workflow member provides overview

2. Participants prepare by developing understanding of document and look for faults

3. Inspection led by one participant through entire document. Written report completed with 1 day

4. Document goes through rework to fix faults

5. Moderator performs follow-up checking that all faults are fixed and documentation (including reports) are updated

Inspection Team

Team consists of 3 - 6 members, including: Moderator Member of team performing current workflow Member of team performing next workflow SQA team member

One person will serve as reader Leads team through the document

One person will serve as recorder Produces report of all possible faults team finds

Metrics Maintained

During inspection, detailed metrics kept Is each fault a minor fault or a major fault

Is the fault from not meeting the specification or that actual & formal arguments do not match?

Where in the project did the fault arise? How many faults were found per document/LOC? How long did it take to find a fault?

Using Metrics

Can compare between products Are we getting better at a workflow? Are the value correlated with something

Take actions if artifact is overly buggy Suggests design may need rethinking

Consider fault statistics from next workflow Some faults may not be detected until later Important to perform quality check on quality team

Testing Results

JPL found savings of $25,000 per inspection Fault counts decreased exponentially by phase

Important to maintain testing independence Benefits tied to faith in testing team Testing metrics should not be used as

performance evaluation Need to consider metrics scientifically

One or two results is not a sufficient sample Statistics can always be bent to tell a story

Inspections vs. Walkthroughs

Walkthrough Two-step, informal process

Inspection Five-step, formal process

Both good for finding faults early in process But rely on having reliable, complete process Requires thorough documentation be available Lastly, needs all teams to buy-in to approach

Correctness Proofs

Halfway between execution-based & non-execution-based testing Examines if algorithm/code matches specification But does so by developing mathematical proof

Relies on creating & understanding proof Also assumes specification is correct

Correctness Proof

Begins by converting code to flowchart

Correctness Proof

Add I/O specification Loop invariants Assertions

Developingthese can beas much art asscience

Three Myths

Software Engineers stink at math Relies on basic algebra; even I passed that class

Costs way, WAY, WAY too much Might be worth it in some situation; use sparingly

Proofs are too darn hard Nontrivial products have been successfully proven Theorem provers can help automate process

Are specification, compiler, & tools correct? Oops, this is real concern

For Next Lecture

Wednesday’s lecture discusses execution testing Book provides good overview of how this is done,

who should do the testing Additional reading presents JUnit testing tool

Before next lecture Spend a few minutes writing a simple class and

then playing with JUnit to test it