Definitions and Concepts of Testing and Quality

What is

Testing?

What is

Quality ?

How Related?

Why Test ?

• Because we can’t assume that the software “works”

– How do we answer the question: “Does this software work?”

• Is it reliable ? (system is continuously functioning > 720hrs)• Is it functional ?

– Complete– Consistent

• Is it responsive ? (response time < 1 second)

• In general, “what is the Quality of our software?”

– How do we answer this?

How would you answer this about the program that you wrote?How would you answer this about the program your friend wrote?

3 major “Testing” approaches

1. Traditionally, testing includes executing the code with test cases.

(assuming - code is the main software artifact)

2. What do we do with the non-executable software artifacts?

Reviews and Inspections

- Expensive in terms of human resources - A lot to maintain and keep updated

3. Can we “prove” that the software works or is defect free?

- Theoretically, given an arbitrary program we can not show that it has no bug.

- We can use Formal Proofs to show the behavior of code

Historical Progression of “Attitudes” on Software Quality

-Large Host & CentralizedSystems-Single Vendor (hdw-sw-serv)-Long term development and long term investment(10 yrs)-Single platform-Systems ran by computer professionals

1980’s & Before

-PC and Desktop Computing became ubiquitous-multiple vendors-quicker product development & shorter term investment-systems ran by non-computer individuals

-Web availability to many-Business conducted on the Web-Software and systems are not hobbies but a “business” again

1990’s

Late 1990’s & 2000’s

**Product Reliability & Quality was required and expected**

**New product was fashionable& “reboot” became acceptable.**

**Product Reliability & Quality is once again important** --especially SECURITY--

Even though we didn’tget good quality

Current State (2000’s) on Testing1. Software is written by many with the

“entrepreneurial” spirit:– Speed to market– New & innovation is treasured– Small organization that can’t afford much more than

“coders”

2. Embracing “Agile” process and mistaking it as “fast production regardless of what”:

– Not much documented (requirements/design)– Hard to test without documented material

3. Lack of Trained/Good/Experienced Testers– Testers are not rewarded “as equals” to designers, but

gaining on programmers

4. Improvement in tools and standards making development easier and less error prone

Why ?

How ?

ISO/IEC 9126 (1994) Quality “model”

• Quality is composed of several characteristics:

– Functionality – Reliability– Usability– Efficiency– Maintainability– Portability

How do you know if it is achieved & how would you “test” for these?

What do these mean ?

Quality

• Quality is a characteristic or attribute

– Needs to be clearly defined and agreed to• May have sub-attributes (e.g. previous page)• Needs specific metrics for each of the sub-attributes

– Needs to be measured with the defined metric(s)• Needs to be tracked and analyzed• Needs to be projected• Needs to be controlled

• Testing and Measurement are two key activities that would help us manage quality.

Explain ----- the relationship ?

Imagine what it is likewithout this.

Software Quality Assurance (QA)

• Software QA focused on 2 main areas

– Software product– Software process

• The focus on the process areas “borrowed” many techniques from the traditional manufacturing area

– Concept of reliability (number of defects, mean time to failure, probability of failure, etc. metrics)

– Concept of process control in terms of looking at “repeatability” of process--- repeatable process produces “similar” product (or controllable results).

emphasis today : e.g. CMM & CMMI

Software Development is NOT Manufacturing (or is it?)

• Software Development is extremely labor intensive– People are not uniform like machines used in

manufacturing.

• Software Development often requires innovation– Every software seems to be a one of its kind

although more and more are becoming “standardized”

– The same set of people do not get to repeatedly develop the exactly same software multiple times.

Improve Documentation (Availability and Presentation Technique)

• Improve the creation and maintenance of documents via an on-line repository or configuration manager

– Centrally protected data– Sharable data– Managed data (data relationships are managed)

• Improve the “usability” and “productivity” by providing more and better visualization of data

– Replace numbers with graphs and figures– Replace words with pictures

track, project, control

Some Talking Definitions(IEEE terminology based)

• Error – A mistake made by a human– The mistake may be in the requirements, design, code, fix,

integration, or install

• Fault– Is a defect or defects in the artifact that resulted from an error– There may be defects caused by errors made that may or may

not be detectable (e.g. error of omission)

• Failure– Is the manifestation of faults when the software is “executed.”

• Running code

• May show up in several places

• May be non-code related (e.g. reference manual)

• Incident – Is the detectable symptom of failures

(Not in the text)

(includes error of omission and “no-code?”)

Note

Example? (bank accnt)

Testing in the Large• Testing is concerned with all, but may not be

able to detect all – Errors– Faults– Failures– Incidents

• Testing utilize the notion of test cases to perform the activities of test(s)– Inspection of non-executables– Executing the code– Analyzing and formally “proving” the non-

executables and the executable in a business workflow (or user) setting

Software Activities and Error Injections, Fault Passing, and Fault Removal

Requirements

Design

Code Testing

Error

Error

Error

Fixing

Error

faults

faults

faults

Inspection

Inspection

Fixing

Note that in “fixing” failures, one can commit errors and introduce faults

Recording each Test Case

• Test Case number• Test Case author • A general description of the test purpose• Pre-condition• Test inputs• Expected outputs (if any)• Post-condition

• Test Case history:– Test execution date

– Test execution person

– Test execution result (s)

Specification vs Implementation

Specification Implementation

Expected Actual

The ideal place where expectation and actual “matches” The other areas are of concern ---- especially to testers

Specification vs Implementation vs Test Cases

Specification Implementation

Expected Actual

Test Cases

Tested

1 32

4

5

6

7

What do thesenumbered regionsmean to you?

Black Box vs White Box testing

• Black box testing (Functional Testing)– Look at only the input and outputs– Mainly uses the specification as the source for

designing test cases. – The internal of the implementation is not included

in the test case design.– Hard to detect “missing” specification

• White box testing (Structural Testing)– Look at the internals of the implementation– Design test cases based on the design and code

implementation– Hard to detect “extraneous” implementation that

was never specified

Fault Classification (Beizer)

• Mild – misspelled word• Moderate - misleading or redundant info• Annoying – truncated names; billing for $0.00• Disturbing – some transactions not processed• Serious - lose a transaction• Very serious - incorrect transaction execution• Extreme – Frequent very serious errors• Intolerable - database corruption• Catastrophic – System shutdown• Infectious - Shutdown that spreads to others

IEEE list of anomalies (faults)

• Input/output faults• Logic faults • Computation faults• Interface faults• Data faults

Why do you care about these “types” of faults (results of errors made)?

Because they give us some ideas of what to look for in designing test cases

Different Levels of Testing

Program unit A

Program unit B

Program unit T

.

.

.

Function 1

Function 2

Function 8

.

.

Component 1

WholeSystem

Component 3

.

Unit Testing Functional Testing Component TestingSystem Testing

Still Need to Demonstrate Value of Testing• Catastrophic problems (e.g. life or business ending ones)

do not need any measurements---but--- others do:

– Measure the cost of problems found by customers• Cost of problem reporting/recording• Cost of problem re-creation• Cost of problem fix and retest• Cost of solution packaging and distribution• Cost of managing the customer problem-to-resolution steps

– Measure the cost of discovering the problems and fixing them prior to release

• Cost of planning reviews and testing• Cost of executing reviews and testing• Cost of fixing the problems found and retest• Cost of inserting fixes and updates• Cost of managing problem-to-resolution steps

– Compare the above two costs AND include loss of customer “good-will”

Goals of Testing?

• Test as much as time allows us– Execute as many test cases as time allows?

• Validate all the “key” areas– Test only the designated “key”requirements?

• Find as much problems as possible – Test all the likely error prone areas and

maximize test problems found?

• Validate the requirements– Test all the requirements?

• To reach a quality target

State your goal(s) for testing. - - - what would you like people to say about your system ? Your goals may dictate your testing process

Very important to think about this ---

Quality Target?