Programming Techniques Lecture 14 Quality Management Based on: Software Engineering, A Practitioner’s Approach, 6/e, R.S. Pressman Software Engineering

Programming Techniques

Lecture 14Quality Management

Based on: Software Engineering, A Practitioner’s Approach, 6/e, R.S. Pressman

Software Engineering

Fall 2005


What is quality?

• Quality, simplistically, means that a product should meet its specification.

• This is problematical for software systems– There is a tension between customer quality requirements

(efficiency, reliability, etc.) and developer quality requirements (maintainability, reusability, etc.);

– Some quality requirements are difficult to specify in an unambiguous way;

– Software specifications are usually incomplete and often inconsistent.


Quality

• The American Heritage Dictionary defines quality as

– “a characteristic or attribute of something.”

• For software, measures of program’s characteristics include:

- cyclomatic complexity, cohesion, number of function points, lines of code.


Quality Concepts

• For software, two kinds of quality may be encountered:

– Quality of design encompasses requirements, specifications, and the design of the system.

– Quality of conformance is an issue focused primarily on implementation.


Quality Concepts

Difference between "quality of design" and "quality of conformance“:

• Quality of design refers to the characteristics that designers specify for the product being constructed.

• Quality of conformance is the degree to which the requirements and design specifications are followed during the manufacturing of the product.


Quality Concepts

• Quality is important, but if the user is not satisfied, nothing else really matters.

user satisfaction = compliant product + good quality + delivery within budget and schedule


Software Quality

• Conformance to explicitly stated functional and Conformance to explicitly stated functional and performance requirements, explicitly performance requirements, explicitly documented development standards, and documented development standards, and implicit characteristics that are expected of all implicit characteristics that are expected of all professionally developed software. professionally developed software.


Quality Concepts

• Quality control - series of inspections, reviews, and tests used to ensure conformance of a work product to its specifications

• Quality assurance - consists of the auditing and reporting procedures used to provide management with data needed to make proactive decisions


Cost of Quality

• Prevention costs include: quality planning, formal technical reviews, test equipment, training.

• Appraisal costs include: in-process and inter-process inspection, equipment calibration and maintenance, testing.

• Internal failure costs include: rework, repair, failure mode analysis.

• External failure costs are: complaint resolution, product return and replacement, help line support, warranty work.


Software quality management

• Concerned with ensuring that the required level of quality is achieved in a software product.

• Involves defining appropriate quality standards and procedures and ensuring that these are followed.

• Should aim to develop a ‘quality culture’ where quality is seen as everyone’s responsibility.


Quality Management System (QMS)

• A quality management system (QMS) is an organization-wide mechanism for building quality into projects and for managing the quality control process. Figure 1 illustrates these basic elements.

• A quality management system will include a quality manual, which in turn references a number of standards, guidelines and procedures that are applied within the organization. The quality manual will require that each project develops its own quality plan, which must comply with the guidelines laid down in the quality manual. The quality management system (QMS) itself is likely to want to claim conformance to some external national or international standard for quality.


Figure 1. The architecture of a quality management system


The quality compromise

• We cannot wait for specifications to improve before paying attention to quality management.

• We must put quality management procedures into place to improve quality in spite of imperfect specification.


Scope of quality management

• Quality management is particularly important for large, complex systems. The quality documentation is a record of progress and supports continuity of development as the development team changes.

• For smaller systems, quality management needs less documentation and should focus on establishing a quality culture.


Role of the Software Quality Assurance (SQA) Group-I

• Prepares an SQA plan for a project.

– The plan identifies» evaluations to be performed

» audits and reviews to be performed

» standards that are applicable to the project

» procedures for error reporting and tracking

» documents to be produced by the SQA group

» amount of feedback provided to the software project team

• Participates in the development of the project’s software process description.

– The SQA group reviews the process description for compliance with organizational policy, internal software standards, externally imposed standards (e.g., ISO-9001), and other parts of the software project plan.


Role of the SQA Group-II• Reviews software engineering activities to verify compliance with the defined software

process. – identifies, documents, and tracks deviations from the process and verifies that corrections have

been made.

• Audits designated software work products to verify compliance with those defined as part of the software process.

– reviews selected work products; identifies, documents, and tracks deviations; verifies that corrections have been made

– periodically reports the results of its work to the project manager.

• Ensures that deviations in software work and work products are documented and handled according to a documented procedure.

• Records any noncompliance and reports to senior management.– Noncompliance items are tracked until they are resolved.


Why SQA Activities Pay Off?cost to findcost to find

and fix a defectand fix a defect

100100

1010

loglogscalescale

11

Req.Req.DesignDesign

codecodetesttest

systemsystemtesttest

fieldfielduseuse

0.750.75 1.001.001.501.50

3.003.00

10.0010.00

60.00-100.0060.00-100.00


Software Reviews

• Purpose is to find errors before they are passed on to another software engineering activity or released to the customer.

• Software engineers (and others) conduct formal technical reviews (FTRs) for software engineers.

• Using formal technical reviews (walkthroughs or inspections) is an effective means for improving software quality.


Bugs, Errors, Defects: Terminology

• The general consensus within the software engineering community is that defects and errors, faults, and bugs are synonymous.

• We will make a distinction between an error and a defect:

- error – a quality problem found before the software is released to end-users;

- defect – a quality problem found only after the software has been released to end-users.


What Are Reviews

• A meeting conducted by technical people for technical people

• A technical assessment of a work product created during the software engineering process

• A software quality assurance mechanism

• A training ground


What Reviews Are Not

• A project summary or progress assessment

• A meeting intended solely to impart information

• A mechanism for political or personal reprisal!


The Players

reviewreviewleaderleader

producerproducer

recorderrecorder reviewerreviewer

standards bearer (SQA)standards bearer (SQA)

maintenance maintenance oracleoracle

user repuser rep


Conducting the Reviewbe prepared—evaluate be prepared—evaluate

product before the reviewproduct before the review

review the product, not review the product, not the producerthe producer

keep your tone mild, ask keep your tone mild, ask questions instead of questions instead of making accusationsmaking accusations

stick to the review agendastick to the review agenda

raise issues, don't resolve themraise issues, don't resolve them

avoid discussions of style—stick to technical avoid discussions of style—stick to technical correctnesscorrectness

schedule reviews as project tasksschedule reviews as project tasks

record and report all review resultsrecord and report all review results

1.1.

2.2.

3.3.

4.4.

5.5.

6.6.

7.7.

8.8.


Formal Technical Reviews (FTRs)

What is it? Why is one conducted?

• An FTR is a software quality control performed by software engineers (and others).

• The purpose of an FTR is to have a group of software engineers examine a discrete work product and determine whether on not the product is free of errors, omissions or inconsistencies.

• Software specifications and standards are used as the review criteria.


FTR Objectives

The objectives of an FTR are:• To uncover errors in function, logic, or implementation for any

representation of the software;• To verify that the software under review meets its requirements;• To ensure that the software has been represented according to

predefined standards;• To achieve software that is developed in a uniform manner;• To make projects more manageable.

• The FTR also serves to promote backup and continuity because a number of people become familiar with parts of the software that they may not have otherwise seen.


FTR Meeting Tasks I

• Involves 3 to 5 people (including reviewers)

• Advance preparation (no more than 2 hours per person) required

• Duration of review meeting should be less than 2 hours


FTR Meeting Tasks II

• Focus of review is on a discrete work product. • Review leader organizes the review meeting at the producer's request.• Reviewers ask questions that enable the producer to discover his or her

own error (the product is under review not the producer). • Producer of the work product walks the reviewers through the product

(alternative: in inspections a "reader" who is not the producer presents the work product) .

• Recorder writes down any significant issues raised during the review • Reviewers decide to accept or reject the work product and whether to

require additional reviews of product or not .


FTR Reporting and Record Keeping I

• During the FTR, a reviewer (the recorder) actively records all issues that have been raised. These are summarised at the end of the review meeting and a review issues list is produced.

• In addition, a formal technical review summary report (a single page form) is completed, which answers:

- What was reviewed?

- Who reviewed it?

- What were the findings and conclusions?


FTR Reporting and Record Keeping II

• It is important to establish a follow-up procedure to ensure that items on the issues list have been properly corrected, otherwise, issues raised can ‘fall between the cracks’.

• One approach is to assign the responsibility for follow-up to the review leader.


FTR Guidelines

• Review the product not the producer. • Seat an agenda and maintain it. • Limit rebuttal and debate. • Enunciate problem area, but don't attempt to solve

every problem noted. • Take written notes. • Limit number of participants and insist on advance

preparation. • Develop a checklist for each product that is likely to be

reviewed. • Allocate resources and schedule time for all reviewers. • Conduct meaningful training for all reviewers. • Review your early reviews,


FTR – Should the Programming Style be Reviewed?

• Assessing style is tricky and can lead to bad feelings if a reviewer is not careful when he/she makes comments concerning style.

• If the producer gets the feeling that the reviewer is saying, "Do it like I do," it is likely that some resentment will arise.

• In general, the review should focus on correctness.


Sample Driven Reviews (SDRs)

• Samples of all software engineering work products are reviewed to determine the most error-prone

• Full FTR resources are focused on the likely error-prone work products based on sampling results


SDRs

• The fraction of the work product that is sampled must:

- be representative of the work product as a whole, and;

- large enough to be meaningful to the reviewer(s) who does the sampling.


SDRs

• The SDRs must attempt to quantify those work products that are primary targets for full FTRs. To accomplish this, the following steps are suggested:1. Inspect a fraction ai of each software work product, i. Record the number of faults, fi found within ai.2. Develop a gross estimate of the number of faults within work product i by multiplying fi by 1/ai.

3. Sort the work products in descending order according to the gross estimate of the number of faults in each.

4. Focus available review resources on those work products that have the highest estimated number of faults.


Metrics Derived from Reviews

inspection time per page of documentationinspection time per page of documentationinspection time per KLOC or FPinspection time per KLOC or FP

errors uncovered per reviewer hourerrors uncovered per reviewer hour

errors uncovered per preparation hourerrors uncovered per preparation hour

errors uncovered per SE task (e.g., design)errors uncovered per SE task (e.g., design)number of minor errors (e.g., typos)number of minor errors (e.g., typos)

number of errors found during preparationnumber of errors found during preparation

number of major errorsnumber of major errors (e.g., nonconformance to req.) (e.g., nonconformance to req.)

inspection effort per KLOC or FPinspection effort per KLOC or FP


Statistical Quality Assurance

Statistical quality assurance implies the following steps:

1. Information about software defects is collected and categorized.

2. Each defect is traced back to its cause. 3. Using the Pareto principle (80% of the defects can be

traced to 20% of the causes) isolate the "vital few" defect causes.

4. Move to correct the problems that caused the defects in the "vital few“.


Statistical SQA

ProductProduct& Process& Process

measurement

... an understanding of how to improve quality ...

Collect information on all defectsFind the causes of the defectsMove to provide fixes for the process


Six-Sigma for Software Engineering I

• Six Sigma is the most widely used strategy for statistical quality assurance in industry today.

• Originally popularized by Motorola in the 1980s.


Six-Sigma for Software Engineering II

• The term “six sigma” is derived from six standard deviations—3.4 instances (defects) per million occurrences—implying an extremely high quality standard.


Six-Sigma for Software Engineering Steps

The Six Sigma methodology defines the following steps:

• Define customer requirements and deliverables and project goals via well-defined methods of customer communication.

• Measure the existing process and its output to determine current quality performance (collect defect metrics).

• Analyze defect metrics and determine the vital few causes.• Improve the process by eliminating the root causes of defects.• Control the process to ensure that future work does not reintroduce the

causes of defects.

Sometimes referred to as DMAIC (define, measure, analyze, improve, and control) method.


Software Reliability

• Defined as the probability of failure free operation of a computer program in a specified environment for a specified time period

• Can be measured directly and estimated using historical and developmental data (unlike many other software quality factors)

• Software reliability problems can usually be traced back to errors in design or implementation.


Software Reliability Illustration

• Program X is estimated to have a reliability of 0.96 over 8 elapsed processing hours.

• In other words, if program X were to be executed 100 times and require a total of 8 hours of processing time (execution time), it is likely to operate correctly (without failure) 96 times.


Failure

• In the context of any discussion of software quality and reliability:

Failure is nonconformance to software requirements.


Software Reliability

• A simple measure of reliability is mean-time-between-failure (MTBF), where

MTBF = MTTF + MTTR

MTTF - mean-time-to-failureMTTR - mean-time-to-repair

• Many argue that MTBF is a far more useful measure than defects/KLOC or defects/FP. An end-user is concerned with failures, not with the total error count. Because each defect contained within a program does not have the same failure rate, the total defect count provides little indication of the reliability of a system.


Software Availability

• Software availability is the probability that a program is operating according to requirements at a given point in time and is defined as

Availability = [MTTF/(MTTF + MTTR)] x 100%

• The MTBF reliability measure is equally sensitive to MTTF and MTTR. The availability measure is somewhat more sensitive to MTTR, an indirect measure of the maintainability of software.


MTBF Criticisms

• For hardware the MTBF concept is based on statistical error data that occurs due to physical wear in a product.

• In general, when a failure does occur in hardware, the failed part is replaced with a spare. However, when an error occurs for software, a design change is made to correct it.

• The change may create side effects that generate other errors. Therefore, the statistical validity of MTBF for software is suspect.


Software Safety

• Software safety is a software quality assurance activity that focuses on the identification and assessment of potential hazards that may affect software negatively and cause an entire system to fail.

• If hazards can be identified early in the software process, software design features can be specified that will either eliminate or control potential hazards.

• Software reliability involves determining the likelihood that a failure will occur, while software safety examines the ways in which failures may result in conditions that can lead to a mishap.


Mistake-Proofing I

• Poka-yoke (mistake-proofing) devices—mechanisms that lead to – the prevention of a potential quality problem before it

occurs or

– the rapid detection of quality problems if they are introduced.

• For example, the ignition switch for a motorcar will not work if an automatic transmission is in gear (a prevention device); a car’s warning beep will sound if the seat belts are not buckled (a detection device).


Mistake-Proofing II

• An effective poka-yoke device exhibits a set of common characteristics: – It is simple and cheap. If a device is too complicated or

expensive, it will not be cost effective.

– It is part of the process. That is, the poka-yoke device is integrated into an engineering activity.

– It is located near the process task where the mistakes occur. Thus, it provides rapid feedback and error correction.

• The poka-yoke scripts


ISO 9000 Quality Standards

• An international set of standards for quality management.

• Applicable to a range of organisations from manufacturing to service industries.

• ISO 9001 applicable to organisations which design, develop and maintain products.

• ISO 9001 is a generic model of the quality process that must be instantiated for each organisation using the standard.


ISO 9000 Quality Standards

For the software industry the relevant standards are:• ISO 9001 Quality Systems – Model for Quality Assurance

in Design, Development, Production, Installation and Servicing. This is a standard which describes the quality system used to support the development of a product which involves design.

• ISO 9000-3. Guidelines for the Application of ISO 9001 to the Development, Supply and Maintenance of Software. This is a specific document which interprets ISO 9001 for the software developer.

• ISO 9004-2. Quality management and Quality System Elements – Part 2. This document provides guidelines for the servicing of software facilities such as user support.


Why is ISO 9000 important

• European Union directive– ISO 9000 certification required by suppliers of “Regulated

Products”

» health, safety, and the environment

– EC has strict corporate liability legislation protecting consumers

• Globalization impact


Why adopt ISO 9000?

• To comply with customers who require ISO 9000

• To sell in the European Union market

• To improve the quality system

• To minimize repetitive auditing by similar and different customers

• To improve subcontractors’ performance


ISO 9000 certification

• Quality standards and procedures should be documented in an organisational quality manual.

• An external body may certify that an organisation’s quality manual conforms to ISO 9000 standards.

• Some customers require suppliers to be ISO 9000 certified although the need for flexibility here is increasingly recognised.


ISO 9001:2000 Standard

• ISO 9001:2000 is the quality assurance standard that applies to software engineering.

• The standard contains 20 requirements that must be present for an effective quality assurance system.

• The requirements delineated by ISO 9001:2000 address topics such as

– management responsibility, quality system, contract review, design control, document and data control, product identification and traceability, process control, inspection and testing, corrective and preventive action, control of quality records, internal quality audits, training, servicing, and statistical techniques.


ISO 9001:2000 Standard – Basic Elements

• Establish the elements of a quality management system.

- Develop, implement, and improve the system.

- Define a policy that emphasizes the importance of the system.

• Document the quality system.– Describe the process.

– Produce an operational manual.

– Develop methods for controlling (updating) documents.

– Establish methods for record keeping.

• Support quality control and assurance.


ISO 9000-3

• Because the ISO 9001:2000 standard is applicable to all engineering disciplines, a special set of ISO guidelines (ISO 9000-3) have been developed to help interpret the standard for use in the software process.

• Industry-wide quality “process”– Not software-specific

• Focused on customer-supplier relationship

• Focused on repeatable success through maintenance of basic quality practices

• Mostly auditor’s judgment used for certification

• “Broadly” covers CMMI levels 1-3


SQA Plan

• The SQA Plan provides a road map for instituting software quality assurance.

• Developed by the SQA group (or the software team if a SQA group does not exist), the plan serves as a template for SQA activities that are instituted for each software project.

• A standard for SQA plans has been published by the IEEE.


SQA Plan

• Management section - describes the place of SQA in the structure of the organization

• Documentation section - describes each work product produced as part of the software process

• Standards, practices, and conventions section - lists all applicable standards/practices applied during the software process and any metrics to be collected as part of the software engineering work

• Reviews and audits section - provides an overview of the approach used in the reviews and audits to be conducted during the project

• Test section - references the test plan and procedure document and defines test record keeping requirements

• Problem reporting and corrective action section - defines procedures for reporting, tracking, and resolving errors or defects, identifies organizational responsibilities for these activities

• Other - tools, SQA methods, change control, record keeping, training, and risk management


Improving Quality of Software Across an Organization

• The most important thing to do:

- institute formal technical reviews.

• After these are working smoothly, any of a number of software quality assurance (SQA) activities might be implemented:

- change control and software configuration management (SCM);

-comprehensive testing methodology;

- SQA audits of documentation and related software.

- also software metrics can help.


Documentation standards

• Particularly important - documents are the tangible manifestation of the software.

• Documentation process standards– Concerned with how documents should be developed,

validated and maintained.

• Document standards– Concerned with document contents, structure, and

appearance.

• Document interchange standards– Concerned with the compatibility of electronic documents.


Document standards

• Document identification standards– How documents are uniquely identified.

• Document structure standards– Standard structure for project documents.

• Document presentation standards– Define fonts and styles, use of logos, etc.

• Document update standards– Define how changes from previous versions are reflected in

a document.


Document interchange standards

• Interchange standards allow electronic documents to be exchanged, mailed, etc.

• Documents are produced using different systems and on different computers. Even when standard tools are used, standards are needed to define conventions for their use e.g. use of style sheets and macros.

• Need for archiving. The lifetime of word processing systems may be much less than the lifetime of the software being documented. An archiving standard may be defined to ensure that the document can be accessed in future.


Summary

• Software quality assurance (SQA) is the concern of every software engineer to reduce cost and improve product time-to-market.

• A Software Quality Assurance Plan is not merely another name for a test plan, though test plans are included in an SQA plan.

• SQA activities are performed on every software project. • Use of metrics is an important part of developing a

strategy to improve the quality of both software processes and work products.


2005F

Instructor

John D Lin - [email protected]

Lectures

Tues & Thurs 10:00-11:30 RM. 100, Lower Block