Quality Assurance

IS 101Y/CMSC 101YNovember 12, 2013

Carolyn SeamanUniversity of Maryland Baltimore County

Data Analysis Assignment Stacked bar charts

1 12 23 34 45 56 67 78 89 1001111221331440

50

100

150

200

250

happiness#hours self care#hours studying in groups

Data Analysis AssignmentAggregate and focus

Abby

Brad

Christ

ina

Damia

n Ed

Fran

k

Greta

Helen

aIa

n

Jose

ph0

10

20

30

40

50

60

70

80

Avg #hours self care

Avg #hours self care

Quality AssuranceRefers to any activity whose goal is to make sure

that the system or application being built is free from error and of high qualityThat it worksThat it solves a problemThat it’s usable

A process view of QAProblem Identification

Analysis

Design

Implementation

QA

Maintenance

SDLC:Systems Development Lifecycle

Installation

CMSC, CMPE, IS

QA includes:Testing

Exercises a system or part of a system to make sure it produces the correct output or behaves in expected ways

Cannot happen until some part of the system is implemented

Reviews and inspectionsVisual “reading” of some artifactCan be done early by reviewing early documents,

e.g. design

Testing TermsCoverage

ideally, testing will exercise the system in all possible ways

not possible, so we use different criteria to judge how well our testing strategy “covers” the system

Test caseconsists of data, procedure, and expected resultrepresents just one situation under which the

system (or some part of it) might run

Test PlanningA test plan includes:

test objectivesschedule and logisticstest strategiestest cases

proceduredataexpected result

procedures for handling problems

Testing Phases

Unit testing - does this piece work by itself?

Integration testing - do these two pieces work together?

System testing - do all the pieces work together?

Alpha acceptance testing - try it out with in-house “users”

Installation testing - can users install it and does it work in their environment?

Beta acceptance testing - try it out with real users

In development/ maintenance organization

In user organization

Testing Techniques

Structural testing techniques“white box” testingbased on statements in the code or individual

hardware and connectionscoverage criteria related to physical parts of

the system tests how a program/system does something

Functional testing techniques“black box” testingbased on input and outputcoverage criteria based on behavior aspectstests the behavior of a system or program

System Testing TechniquesGoal is to evaluate the system as a whole, not

its parts

Techniques can be structural or functional

Techniques can be used in any stage that tests the system as a whole (acceptance, installation, etc.)

Techniques not mutually exclusive

System Testing Techniques (cont.)

stress testing - test larger-than-normal capacity in terms of transactions, data, users, speed, etc.

execution testing - test performance in terms of speed, precision, etc.

recovery testing - test how the system recovers from a disaster, how it handles corrupted data, etc.

operations testing - test how the system fits in with existing operations and procedures in the user organization

compliance testing - test adherence to standardssecurity testing - test security requirements

System Testing Techniques (cont.)

requirements testing - fundamental form of testing - makes sure the system does what it’s required to do

regression testing - make sure unchanged functionality remains unchanged

error-handling testing - test required error-handling functions (usually user error)

manual-support testing - test that the system can be used properly - includes user documentation

historical test data - tests until the number of defects found approaches the average number of defects in the products produced under similar circumstances

Unit TestingGoal is to evaluate some piece (file, program,

module, component, etc.) in isolation

Techniques can be structural or functional

In practice, it’s usually ad-hoc and looks a lot like debugging

More structured approaches exist

Unit Testing Techniques

input domain testing - pick test cases representative of the range of allowable input, including high, low, and average values

equivalence partitioning - partition the range of allowable input so that the program is expected to behave similarly for all inputs in a given partition, then pick a test case from each partition

boundary value - choose test cases with input values at the boundary (both inside and outside) of the allowable range

Unit Testing Techniques (cont.)

statement testing - ensure the set of test cases exercises every statement at least once

branch testing - each branch of an if/then statement is exercised

path testing - every path is exercised (impossible in practice)

fault seeding - put a certain number of known faults into the code, then test until they are all found

Reviews and Inspection:Goal and Motivation

By detecting defects early, and preventing their leakage downstream, the higher cost of later detection and rework is eliminated.

Basic StepsUsing a reading technique,

Perform a visual examination of the document

Detect and correct:• Defects• Violation of design standards• Other problems

Examples of artifacts that can be reviewed

Include:ContractsInstallation plansProgress reportsDesign descriptionsRelease notesRequirements specificationsSource code

What Is a Defect?Any occurrence in a work product that is

determined to be incomplete, incorrect, or missing

Any instance which a requirement is not satisfied (Fagan, 1986)

Informal synonyms: bug, fault, issue, problem, anomaly

Common Attributes of Systematic Reviews and

InspectionsSystematic reviews and inspections have these attributes in common:• Team participation• Documented results of the review• Documented procedures for conducting the review

Management Reviews

Performed by those directly responsible for the system

Monitor progress

Determine status of plans and schedules

Confirm requirements and their system allocation

Or, evaluate management approaches used to achieve fitness or purpose

Technical Reviews

Confirms that a productConforms to specificationsAdheres to regulations, standards, guidelines,

plansChanges are properly implementedChanges affect only those system areas

identified by the change specification

Inspections (Formal Peer Reviews)

Confirms that the software product satisfiesSpecificationsSpecified quality attributesRegulations, standards, guidelines, plans Identifies deviations from standards and

specificationresults in logging a defect

Walk-throughsEvaluate a product or artifact or document

Sometimes used for educating an audience

Major objectives:• Find anomalies• Improve the product• Consider alternative implementations• Evaluate performance to standards and specs

Audits

The purpose of an audit is to provide an independent evaluation of conformance of products and processes to applicableRegulationsStandardsGuidelinesPlansProcedures

Inspection Process Most popular is the Fagan method

Inspection is separated into 5/6 phases (Planning) Overview Preparation Inspection Meeting Rework Follow-up

Review and Inspection Pitfalls

Insufficient Preparation

Moderator Domination

Incorrect Review Rate

Ego-involvement and Personality Conflict

Issue Resolution and Meeting Digression

Recording Difficulties and Clerical Overhead

Other QA ActivitiesProcess conformance

Making sure that quality procedures are followed

Risk analysisPlanning for adverse eventsMaking the unexpected expected

Hiring and trainingDeveloping guidelines and standards Identifying new training needs

Project demos This Thursday!!!

No slides

Everyone must participate and answer questions

Show your program Running Code

Talk about what it does and what it doesn’t do YET

10 minutes, including questions

Turn in .pde file and document BEFORE class on Blackboard

Peer EvaluationsTake a look at the Peer Evaluation form and

think about the following two options:

1. Each student gets the actual sheets filled out by their teammates

2. Each student gets a summary of the feedback written by the instructor

Evaluations due in class on THURSDAY (rewards to those who bring them to class)