Responsible Organization

Common Common Supplier Supplier

Common Supplier

Common Supplier

TECHNICAL REPORTS SERIES No. 237

Manual on Quality Assurance Programme Auditing

J I N T E R N A T I O N A L A T O M I C ENERGY A G E N C Y , V I E N N A , 1 9 8 4

MANUAL ON QUALITY ASSURANCE PROGRAMME AUDITING

T h e fo l lowing S ta tes are Member s of t he I n t e r n a t i o n a l A t o m i c Energy Agency:

A F G H A N I S T A N A L B A N I A A L G E R I A A R G E N T I N A A U S T R A L I A A U S T R I A B A N G L A D E S H B E L G I U M B O L I V I A B R A Z I L B U L G A R I A B U R M A B Y E L O R U S S I A N S O V I E T

S O C I A L I S T R E P U B L I C C A M E R O O N C A N A D A C H I L E C H I N A C O L O M B I A C O S T A R I C A CUBA C Y P R U S C Z E C H O S L O V A K I A D E M O C R A T I C K A M P U C H E A D E M O C R A T I C P E O P L E ' S

R E P U B L I C O F K O R E A D E N M A R K D O M I N I C A N R E P U B L I C E C U A D O R E G Y P T E L S A L V A D O R E T H I O P I A F I N L A N D F R A N C E G A B O N G E R M A N D E M O C R A T I C R E P U B L I C G E R M A N Y , F E D E R A L R E P U B L I C O F G H A N A G R E E C E G U A T E M A L A

H A I T I H O L Y SEE H U N G A R Y I C E L A N D INDIA I N D O N E S I A I R A N , ISLAMIC R E P U B L I C O F I R A Q I R E L A N D I S R A E L I T A L Y I V O R Y C O A S T J A M A I C A J A P A N J O R D A N K E N Y A K O R E A , R E P U B L I C O F K U W A I T L E B A N O N L I B E R I A L I B Y A N A R A B J A M A H I R I Y A L I E C H T E N S T E I N L U X E M B O U R G M A D A G A S C A R M A L A Y S I A MALI M A U R I T I U S M E X I C O MONACO M O N G O L I A M O R O C C O NAMIBIA N E T H E R L A N D S NEW Z E A L A N D N I C A R A G U A N I G E R N I G E R I A N O R W A Y P A K I S T A N P A N A M A

P A R A G U A Y P E R U PHILIPPINES P O L A N D P O R T U G A L Q A T A R R O M A N I A SAUDI A R A B I A S E N E G A L S I E R R A L E O N E S I N G A P O R E S O U T H A F R I C A SPAIN S R I L A N K A S U D A N SWEDEN S W I T Z E R L A N D S Y R I A N A R A B R E P U B L I C T H A I L A N D T U N I S I A T U R K E Y U G A N D A U K R A I N I A N S O V I E T S O C I A L I S T

R E P U B L I C U N I O N O F S O V I E T SOCIALIST

R E P U B L I C S U N I T E D A R A B E M I R A T E S U N I T E D K I N G D O M O F G R E A T

B R I T A I N A N D N O R T H E R N I R E L A N D

U N I T E D R E P U B L I C O F T A N Z A N I A

U N I T E D S T A T E S O F A M E R I C A U R U G U A Y V E N E Z U E L A V I E T NAM Y U G O S L A V I A Z A I R E ZAMBIA

The Agency ' s S t a t u t e was app roved on 23 O c t o b e r 1956 by the C o n f e r e n c e on the S t a t u t e of the I A E A he ld at Un i t ed Na t ions H e a d q u a r t e r s , New Y o r k ; it e n t e r e d i n t o f o r c e on 29 July 1957 . T h e Headqua r t e r s of the Agency are s i tua ted in Vienna . Its pr incipal objec t ive is " t o accelera te and enlarge the c o n t r i b u t i o n of a t o m i c energy t o peace , hea l th and p rosper i ty t h r o u g h o u t t he w o r l d " .

© IAEA, 1984

Permission t o r e p r o d u c e or t rans la te the i n f o r m a t i o n c o n t a i n e d in this publ ica t ion m a y be o b t a i n e d by wri t ing t o t he I n t e r n a t i o n a l A t o m i c Energy Agency , Wagramers t rasse 5, P .O. B o x 100, A-1400 Vienna," Aust r ia .

Pr in ted by the I A E A in Austr ia July 1984

TECHNICAL REPORTS SERIES No. 237

MANUAL ON QUALITY ASSURANCE PROGRAMME AUDITING

INTERNATIONAL ATOMIC ENERGY AGENCY VIENNA, 1984

MANUAL ON QUALITY ASSURANCE PROGRAMME AUDITING IAEA, VIENNA, 1984

STI/DOC/10/237 ISBN 92—0—155184—3

FOREWORD

The Agency's plans for establishing safety standards for nuclear power plants, referred to as the NUSS Programme, include the development of three types of documents:

(1) Codes of Practice for thermal neutron nuclear power plants that establish the objectives and minimum requirements which must be fulfilled to provide adequate safety for these plants

(2) Safety Guides that provide additional requirements and recommend pro-cedures that should be followed to implement the Codes of Practice

(3) User's Manuals, directed primarily to nuclear power plant operators, that normally present possible methods and techniques for solving specific problems

Work on Codes and Guides was initiated in 1975 in five main fields: govern-ment organization, siting, design, operation and quality assurance.

In the field of quality assurance the Code of Practice and eleven Safety Guides had been developed by 1983 and published in English, French, Spanish and Russian. These documents are now used in a number of Member States as quality assurance requirements for nuclear power plants. To facilitate this use the Technical Review Committee on Quality Assurance has stressed on a number of occasions the need for and importance of proceeding with the development of User's Manuals. These documents should provide Member States implementing the Code and the Safety Guides with practical examples of procedures, practices and documents illustrating quality assurance methods and techniques used in those organizations in Member States having broad experience in quality assurance. The same opinion was expressed in the discussions during the Inter-national Symposium on Quality Assurance for Nuclear Power Plants held in Paris in May 1981. A number of topics have been identified for which User's Manuals could provide additional information and facilitate correct implementa-tion of the Code and Guides in nuclear power plant project activities.

To implement these recommendations, work has been initiated in the Secretariat to develop those User's Manuals which are most needed in Member States embarking on nuclear power programmes and starting quality assurance activities. Keeping in mind the difference between User's Manuals and Codes and Safety Guides, work on development of these documents is undertaken out-side the NUSS Programme and the established procedures for development, review and approval of the documents used in this Programme. For User's

Manuals it was decided to follow the standard practices used in the development of the Agency's publications such as Guidebooks and Technical Reports. This procedure will reduce the time and cost of preparation of User's Manuals, which are at the lower level in the hierarchy of NUSS Programme documents and do not contain requirements for whose formulation a broad consensus of QA experts would be needed.

However, it will be ensured that the Manuals are fully consistent with the Code and Safety Guides through information exchange between the Secretariat and members of the Technical Review Committee on Quality Assurance and the Senior Advisory Group of the NUSS Programme. Also, members of these two Committees in the capacity of consultants to the Agency and members of Advisory Groups will be engaged in the development, review and approval of the Manuals before the documents are recommended to the Director General for publication.

The present Manual on Quality Assurance Programme Auditing contains supporting material and illustrative examples for implementing requirements for quality assurance programme audits as stated in the Code of Practice on Quality Assurance for Safety in Nuclear Power Plants, IAEA Safety Series No. 50-C-QA, Section 13, and additional requirements and recommendations presented in the Safety Guide on Quality Assurance Auditing for Nuclear Power Plants, IAEA Safety Series No. 50-SG-QA10. This Manual is directed primarily towards quality assurance programme auditors and managers and it presents methods and techniques considered appropriate for the preparation and per-formance of audits and the evaluation of results.

CONTENTS

1. INTRODUCTION 1

1.1. Manual objective and use 1 1.2. Purpose of audits 1 1.3. Responsibility for auditing 2

2. AUDIT MANAGEMENT 4

2.1. Policies and organization 4

2.2. Procedures 5

3. PRE-AUDIT ACTIVITIES 6

3.1. Overall planning and scheduling 6

3.2. Schedule review 7 3.3. Individual audit plan 7

3.3.1. Preparation 7 3.3.2. Definition of purpose and scope 8 3.3.3. Schedule for individual audit 8 3.3.4. Identification of performance standards 9 3.3.5. Preparation of check-lists and procedures 10 3.3.6. Audit plan documentation 11

3.4. Audit team organization and preparation 12 3.4.1. Personnel requirements 13 3.4.2. Audit team preparation 13

3.5. Audit notification and initial contact with auditee 14 3.6. Pre-audit conference 15

4. AUDIT PERFORMANCE 17

4.1. Audit techniques 17 4.1.1. Sampling 17 4.1.2. Reviewing documents 18 4.1.3. Interviewing 18 4.1.4. Witnessing implementation 19 4.1.5. Tracing 19 4.1.6. Corroboration 20

4.2. Recording audit findings 20 4.3. Audit team co-ordination 21

5. POST-AUDIT ACTIVITIES 21

5.1. Post-audit conference 22 5.2. Reporting 23 5.3. Response 24 5.4. Follow-up 25

6. RECORDS 25

ANNEX: EXAMPLES OF AUDIT DOCUMENTS 27

Example 1. Extract from a quality assurance programme description 28

Example 2. Conduct of quality assurance department audits 29 Example 3. Scheduling of quality assurance department audit activities 37 Example 4. Internal audit schedule 44 Example 5. Audit project reminder check-list 45 Example 6. Audit plan 46 Example 7. Individual audit schedule 47 Example 8. Audit agenda 48 Example 9. Supplier QA system evaluation check-list 49 Example 10. Quality assurance audit check-list 75 Example 11. Supplier evaluation check-list 110 Example 12. Check-list for auditing specific activity 120 Example 13. Check-list for auditing specific activity

(Document distribution) 125 Example 14. Audit procedure 127 Example 15. Lead auditor qualification record 129 Example 16. Qualification of audit personnel 130 Example 17. Check-list based on tracing technique 131 Example 18. Check-list for corroboration 132 Example 19. Finding recording form 133 Example 20. Audit finding form 134 Example 21. Audit report form 135 Example 22. Audit report: transmittal letter and audit report 137 Example 23. Audit report: transmittal letter, audit report finding, and

response letter 143

DRAFTING AND REVIEWING BODIES 149

1. INTRODUCTION

1.1 Manual objective and use

The IAEA Code of Practice on Quality Assurance for Safety in Nuclear Power Plants, hereafter referred to as the Code, establishes requirements for implementing a system of planned and documented internal and external audits to verify compliance with all aspects of a quality assurance (QA) programme and to determine its effectiveness. These requirements are further elaborated in the Safety Guide on Quality Assurance Auditing for Nuclear Power Plants (IAEA Safety Series NO.50-SG-QA10), which contains requirements and recommendations on the management, preparation and performance of QA programme audits.

The objective of this Manual is to provide guidance and illustrative examples of the methodology and techniques of internal and external audits that are con-sistent with the requirements and recommendations of the Code and the Safety Guide. The methodology and techniques are based on the practices of Member States having considerable experience in auditing QA programmes. This Manual is directed primarily towards QA programme auditors and managers and presents methods and techniques considered appropriate for the preparation and per-formance of audits and the evaluation of results. Its scope includes the techniques and methods used to carry out QA programme audits variously described as 'System', 'Product ' and 'Process' audits. The techniques and methods described here may be used as one approach to the evaluation of suppliers' QA capabilities as defined in 50-SG-QA10.

Although the Manual is primarily directed towards purchasers and suppliers, it is also relevant to regulatory organizations, such as government offices responsible for quality assurance, which carry out external audits independent of purchasers and suppliers. In such cases similar methods, procedures and techniques may be used.

1.2. Purpose of audits

According to the Code an audit is "A documented activity performed to determine by investigation, examination and evaluation of objective evidence the adequacy of, and adherence to, established procedures, instructions, specifications, codes, standards, administrative or operational programmes and other applicable documents and the effectiveness of implementation".

This broad definition, when applied to a QA programme audit, implies that an audit (1) represents a documented activity, (2) uses methods of evaluation, examination and investigation of objective evidences, and (3) has the objective of verifying the adequacy and compliance of a QA programme with specified require-ments and determining its effectiveness.

1

According to the Code an overall QA programme must be established as an integral part of a nuclear power project. The responsible organization (as defined in the Code) is responsible for establishing and implementing this programme. This organization may delegate to others the work of establishing and implementing all or part of the QA programme but must retain the responsibility for ensuring its effectiveness in complying with regulatory requirements. Similarly, any organization having delegated technical responsibility for a work scope in a nuclear power project, including pertinent actions required by the overall quality assurance programme, may further delegate to a lower-tier organization a part of that work scope and the appropriate quality assurance actions. An organization verifies the effectiveness of its own QA programme and the effectiveness of that portion which is delegated to other organizations by performing internal or external QA programme audits, respectively. Regardless of whether an audit is performed internally or externally the objectives can be summarized as follows:

(1) To determine that an adequate QA programme exists and that QA procedures have been developed and documented;

(2) To determine, by examination of objective evidence, that the QA programme is being implemented in compliance with specific requirements;

(3) To evaluate the effectiveness of the programme; (4) To identify deficiencies and non-conformances; (5) To recommend corrective action;

(6) To provide management of the audited organization with an assessment of the status and adequacy of the QA programme.

1.3. Responsibility for auditing

The responsible organization is ultimately accountable for every aspect of a nuclear power plant because it is the applicant for a construction permit and lincensee for reactor operation. This responsibility stems from regulatory require-ments for safety and includes establishing and implementing the overall QA programme and the auditing function, which is a major element of the QA programme. This system encompasses the whole hierarchical chain of nuclear power project participants and includes all purchasers and suppliers of items and services, as indicated in Fig. 1.

In this system an internal audit encompasses those portions of an organization's QA programme retained under its direct control. An external audit covers those portions which are not retained under its direct control and not within its organizational structure.

The responsibility of each participant in this chain may be a consequence of direct regulatory requirements and/or of contractual requirements. Contract requirements are normally specified by referencè to the applicable Codes or Standards.

2

Respons ib le Organ iza t ion

Common Common Common Common

Supplier Supplier Supplier Supplier

FIC.l. Purchasers in the hierarchical chain are normally responsible for audits of suppliers.

However, when considered necessary to verify the effectiveness of the QA programme for

which he is responsible, the purchaser may also perform an audit of the lower tier supplier or

participate in external audits performed by his supplier.

Where audits are being carried out to evaluate suppliers, the extent and nature of the auditing organization's activities may vary depending on such considerations as:

(1) Previous third-party audit for the purpose of evaluation of an organization's generic QA programmes. There are many national and international schemes for accrediting these programmes of suppliers of items and services, e.g. ASME1, AFCEN2, etc. In scheduling audits and developing specific audit plans due account should be taken of current supplier accreditation.

1 American Society of Mechanical Engineers. 2 Association française pour les règles de conception et de construction des matériels

des chaudières électro-nucléaires.

3

(2) The likely requirements of other purchasers involved in the nuclear project. There are economic advantages to be gained by collaboration between purchasers in the evaluation of potential suppliers, e.g. purchasers A and B, both involved in a nuclear project, identify common suppliers and perform collaborative audits. Responsibilities of all parties involved in collaborative audits must be established and agreements reached with respect to selection of team leaders and team members.

(3) The availability of information from formal collaborative agreements such as CASE.3

2. AUDIT MANAGEMENT

2.1. Policies and organization

The requirements for QA programme audits are implemented through an organization's policies and procedures established at levels consistent with the organization's scope of work and responsibilities. The highest level policy relevant to the audit programme might consist of a commitment to an effective QA programme consistent with a regulatory requirement or an industrial standard or both. Such a policy may be included in a management policy guide or other top level organizational document and should at least be referenced in the next hierarchical level of policy statement.

The next hierarchical level of policy may be documented in a QA programme description or manual and may be a policy statement with a commitment to establish a system of internal and external audits consistent with regulatory require-ments, applicable codes and standards, and the organization's delegated programme responsibility. When QA requirements are based on the Code, the policy statement in the programme description may have a form such as given in Example 1 of the Annex. Thus, the policy constitutes a commitment by the organization's senior management to establish an independent system of audits which will be used by the management as a tool to reveal and correct possible deficiencies in the QA programme.

The responsibility for carrying out the audit policy may be delegated to a specific audit group or it may be the responsibility of an ad hoc team of qualified individuals from various organizational units. Regardless of how each audit team is made up, there should be a group with responsibility for the audit which reports to the person or organization responsible for ensuring the effectiveness of the QA programme. This implies that the basic principles for the organization of the QA functions as outlined in the Code should be observed when establishing audit team authority, responsibility and operating procedures. As a group having responsibility

3 Co-ordinating Agency for Supplier Evaluation.

4

for quality verification functions, the audit team must be independent of the group it audits and shall report at the same level, or higher, than the organization, funct ion or group it audits. This organizational independence applies also to each individual auditor, who must be sufficiently independent from the activities and functions being audited including cost and schedule and be able to report facts as he observes them. For this reason, he must not report to a supervisor who is directly responsible for activities that he must examine.

This organizational independence of the audit group and of each individual audit member shall be clearly defined and documented. This is normally done through development of an organization chart and the associated descriptions of responsibilities which are part of the audit management procedure.

2.2. Procedures

The QA auditing policy requires the development of written procedures to describe methodology. There may be two types of procedures related to QA audits: the first is a general type of programmatic procedures applicable to all units; the second includes implementing procedures prepared for the individual audit.

Programmatic procedures provide directives for preparing and performing audits in a planned and systematic manner for both external and internal audits. Such procedures are primarily concerned with management actions and provide requirements for the responsibility of groups and persons and proceed with specifications of the sequence of events to be followed in preparing, performing, documenting results, and reporting audits. These procedures also specify scheduling and planning activities to be followed.

The Annex provides examples of programmatic procedures of an organization having overall responsibility for plant construction. In this specific case there are two programmatic procedures which are a part of the Department of Quality Assurance Procedure Manual:

Proc. No. QP 18.1, Conduct of Quality Assurance Department Quality Audits4

(Example 2)

Proc. No. QA 18.2, Scheduling of Quality Assurance Department Audit Activities (Example 3)

Implementing procedures or work instructions for auditors normally deal with individual audits. The Code requires that audits be performed in accordance with written procedures or check-lists. It indicates that implementing procedures

4 Quality Audit is an expression used in some organizations for Quality Assurance Audit.

5

may be considered as an alternative to audit check-lists and that procedures and check-lists are interchangeable. Additional information on implementing procedures is presented in Section 3.3.5.

3. PRE-AUDIT ACTIVITIES

3.1. Overall planning and scheduling

Pre-audit activities normally begin with planning and scheduling: planning includes the systematic determination of what should be done and scheduling establishes the detailed timetable. Requirements for planning and scheduling are provided by QA procedures. These requirements normally include: (1 ) responsibility of respective groups or individuals for initiation of audit planning and scheduling; (2) preparation of the overall schedule to provide for the audit of the complete quality assurance programme and for its constituent elements; (3) topic of each individual audit; and (3) frequency of audits.

In the preparation of an overall audit plan and schedule, internal and external audits are treated separately. Therefore, two separate plans and schedules should be prepared. However, it is necessary to relate the two plans and schedules to optimize the manpower resources required and to provide smooth development of nuclear project activities.

The Safety Guide on Quality Assurance Auditing for Nuclear Power Plants, 50-SG-QA10, specifies conditions for scheduling audits. The conditions applicable to internal audits include:

(1 ) when a systematic independent assessment of QA programme effectiveness is considered necessary;

(2) when significant changes are made in functional areas of the QA programme, such as significant reorganization or revisions of the procedures;

(3) when it is suspected that the quality of an item or service is in jeopardy because of a deficiency, either in requirements or implementation of the QA programme; or

(4) when it is necessary to verify implementation of required corrective actions.

In addition, for scheduling external audits, the following conditions are relevant:

(1) when it is necessary to determine the capability of a supplier's QA programme before awarding a contract or purchase order; and

(2) when, after a contract has been awarded, sufficient time has elapsed for implementing the programme and it is appropriate to determine whether the organization is adequately performing the functions as defined in the programme, applicable codes and standards, and other contract documents.

6

The overall plan and schedule for internal audits should include: (1 ) organi-zational unit to be audited, respective work area or the whole organization; (2) type of audit to be performed (implementation, follow-up, etc.); (3) f requency; (4) tentative dates; (5) reference to documents f rom previous audits, QA programme, procedures or other documents to be used; and (6) estimated manpower requirements.

For external audits the overall plan and schedule should include in addition: (1) organization to be audited; (2) type of audit (evaluation, performance); (3) audit notification (who should be notified and when); and (4) reference to contractual requirements.

Overall schedules may be prepared in the form of a master chart or of a descriptive text. Annex Example 4 provides examples of overall schedules for internal audits. In general the overall audit schedule does not include the sequence of activities to be performed or the scope of each audit. These are to be presented in detail in each individual audit plan.

Consideration should be given to conducting frequent audits during the early stages of an activity, which provides opportunities for discovering deficiencies and implementing corrective action before the activity becomes too advanced.

3.2. Schedule review

The overall audit schedule should be discussed with the audited organizations or their representatives and should be reviewed periodically and revised as necessary to reflect current programme activities.

To be effective, an audit should be performed in an optimal condition with respect to availability of staff of the auditee's organization and in accordance with the performance of activities to be audited. It is therefore important to establish a mutually satisfactory schedule and suggested changes of dates should be considered. However, any suggestions related to the audit scope which might compromise the objective of the audit should be rejected.

3.3. Individual audit plan

3.3.1. Preparation

For each individual audit a plan should be prepared, should be in conformance with the established overall audit plan and schedule, and should include the following elements: scope, requirements, composition of the audit team, activities to be audited, organizations to be notified, applicable documents, schedule, written procedures or check-lists, and language in which the audits will be conducted.

The preparation of an individual audit plan is the responsibility of the auditing organization and is generally carried out by a lead auditor. It is customary for individual audits to be prepared using a check-list; as a minimum, an audit plan

7

reminder should be prepared which lists the steps to be accomplished during the preparatory phase. An example of such a check-list is given in the Annex, Example 5, and a sample audit plan is given in Example 6.

3.3.2. Definition of purpose and scope

The Code establishes requirements for audits to determine the existence, adequacy, effectiveness and correct implementation of the QA programme for a nuclear power plant. Although there are audits with other purposes, only auditing of a QA programme is considered in this document . In the framework of this general purpose an individual audit may have different scopes. It is necessary for the scope of each audit to be established and for the area to be audited to be defined. The scope of an audit may be one of the following: (1) The entire Q A programme of the organization. This type of audit may be

limited in depth, but should cover all elements of the programme. (2) A consti tuent activity in the QA programme such as design or procurement.

This type of audit covers all elements of the selected activity and may be irrespective of the various QA programmes or projects where they are implemented.

(3) Specific practices related to an activity, e.g. design change control or acceptance of procured items. This can be the scope of a follow-up audit when deficiencies in these activities are identified. All these types of audits should be based on specific requirements. In defining

the scope of an audit these requirements should be used as a basis for defining criteria against which the audited quality assurance programme or its constituent elements will be assessed. In accordance with the definition each audit evaluates objective evidence which is then compared with established criteria. The scope of an audit should identify the objective evidences to be examined and to what extent. In this respect the following should be considered in determining such evidence: (1 ) documentat ion, such as programmes, procedures and quality assurance records; (2) organizational documents including authori ty, responsibility and lines of co-ordination and information exchange; (3) performance of activities; (4) items and services; and (5) information obtained during interview. The scope of an audit with respect to choice of objective evidence and methodology should be predetermined and defined during audit preparation.

3.3.3. Schedule for individual audit

Every individual audit plan should include a schedule of events. A detailed schedule is of particular significance for external audits where efficient co-operation should be established between the auditee and the auditing team. In establishing a schedule the following points should be considered: (1) time for the pre-audit conference; (2) approximate timing for each portion of the audit;

8

(3) times for audit team meetings and consultation; (4) titles of auditee personnel expected to be interviewed and involved in the pre- and post-audit conferences; (5) time for the post-audit conference; and (6) ways the schedule may be changed by unforeseen developments.

To assist in the preparation of a detailed schedule the auditing personnel should determine and commit the availability of interface personnel and their alternates. The audit schedule should be documented, either in the form of a master chart or in the form of an agenda. Examples of individual audit schedules presented as a master chart or as an agenda are given in the Annex, Examples 7 and 8 (from 50-SG-QA10).

3.3.4. Identification of performance standards

An important element of preparation is the identification of the performance standards or criteria against which the adequacy and effectiveness of the audited QA programme will be assessed. The audit requirements are specified in the Code and in national regulations or industrial standards. In addition to these general requirements, activities of an audited organization are based on a defined set of governmental regulations, codes, standards, internal policies and procedures, and drawings and specifications. All such documents are considered to be performance standards and should be identified as appropriate in the course of audit preparation.

Depending on the type and scope of the audit, appropriate performance standards should be selected. When the overall QA programme is audited the applicable standard, such as the Code, should be used. The adequacy of established procedures and instructions should be measured against the organization's QA programme and respective commitments in its policies. When the audit is oriented towards activities or towards items and services, the relevant industrial standards, work procedures or instructions should be used.

Other information which might give insight into the auditee's activities and QA programme should also be identified and reviewed: (1 ) all applicable regulatory documents such as safety guides, recommendations, etc.; (2) commitments included in safety analysis reports; and (3) lists and descriptions of inspections and tests performed by the auditee including the description and specification of inspection, testing and measuring equipment.

In the main, the effectiveness of a QA programme will be judged on the basis of qualitative criteria and performance standards; it is essential that there be a common understanding of these standards within the audit team. However, where possible, quantitative criteria should be used to facilitate and contribute to the measurement of the programme's effectiveness. Statistical sampling of documents for examination is a measurement approach using criteria of this type.

9

3.3.5. Preparation of check-lists and procedures

Check-lists and procedures should be developed to ensure that all applicable requirements of performance standards are considered and to provide a means for recording the results of each auditor's examination. A check-list is intended as a guide and should not restrict the investigation; auditors should always look for good, or deficient, activities outside the prepared check-lists.

There are no standard check-lists except those used by some organizations for evaluation of a contractor 's overall QA programme. General check-lists may also be prepared by the auditing organization, although they are normally customized in the preparation of the individual audits. The form and content of check-lists prepared for an individual audit depend on several factors, such as type of audit, scope, and personal preferences of the auditor. Although a check-lists should satisfy the auditor 's needs, there are nevertheless certain criteria which every check-list should include:

(1) Provision for identification of the audit to which it applies; the activity or organization to be audited; audit dates; standards against which the audited QA programme will be evaluated

(2) The name(s) of the auditor(s) (3) Definition of specific performance standards including cross-reference to the

specific section of standards or QA programme documents defining the requirements

(4) Space for recording the results of the auditor 's examination, which may be given in a single block to provide for a summary of results or may include space for recording the results of each step

(5) Identification of specific supporting documentation, such as: a detailed plan for collecting specific evidence to answer each check-list questions, auditor data sheets, list of persons to be contacted, and findings sheets

(6) Auditor 's conclusions

There is a tendency to develop a set of standard check-lists to evaluate an overall QA programme. This is useful when many audits with the same purpose are to be performed. This is the situation with specialized agencies performing audits of several organizations for common use, such as CASE in the USA or EDF in France. The CASE check-lists are based on requirements of the 10 CFR 50 App. B and associated ANSI Standards and those of the EDF on the IAEA Code. Examples are given in the Annex, Examples 9 and 10.

In preparing a check-list for an individual audit, the most important point is to list specific requirements derived from the performance standards or other information. This part of a check-list may be prepared in the form of a statement of requirements or alternatively in the form of appropriate questions. In the latter

10

case it is expected that the answer will be yes or no, indicating compliance or non-compliance. The advantage of using questions is that each requirement paragraph can be divided into a few simple ones to be answered by the auditor himself. A set of questions for a specific requirement allows the auditor to make his own judgement based on objective evidence. Since this judgement should be objective, questions in the audit check-list should not be biased by the auditor 's personal interpretation of requirements or by his preferred way of accomplishing a task. The audit team should have a common understanding of the interpretation of requirements and should have participated in the preparation or review of any check-list questions specific to the individual audit. This is particularly important when qualitative performance standards are used which may contain non-quantifiable attributes, such as 'significant' (deviation or change), 'appropriate ' (action) or 'as necessary', 'when required', etc.

The Annex, in addition to the previously cited examples, gives some fur ther examples of various forms and types of audit check-lists (11, 12, 13).

In Section 2.2 the use of implementing procedures instead of audit check-lists is discussed. It is the auditing organization's responsibility to decide when audit procedures may be more useful than a check-list unless otherwise specified in the QA programme or overall audit plan and schedule. Examples of when audit procedures may be more appropriate than check-lists are: (1) when a member of the audit team is relatively inexperienced and requires more detailed guidance in performing tasks; (2) when a member, or members, are technical specialists without prior auditing experience; and (3) when a complex activity is being audited and a step-by-step procedure would facilitate the performance of the audit. Example 14 of the Annex provides a sample procedure related to an audit of a nuclear power plant documentat ion centre.

3.3.6. Audit plan documentation

All elements of the pre-audit activities should be documented in the form of an individual audit plan, which should summarize conveniently the documents determined during pre-audit activities to be relevant. An individual formal plan is a document which identifies the audit to be performed and specifies the sequence of events. It includes the following elements:

(1 ) Audit identification: the assigned audit number, the subject of the audit and the date of preparation of the plan.

(2) Audit authori ty: e.g. the organization or agency for which the audit is performed." For internal audits the authori ty is the organization's top management and the audit is carried out under the authority of the policy statement of this organization.

11

(3) Purpose of the audit: e.g. the determination of overall Q A programme effectiveness or whether a specific activity is performed correctly. The scope of the audit should be presented in sufficient detail to enable the audit to proceed smoothly.

(4) Identification: e.g. of performance standards or relevant requirements against which the programme will be evaluated. It is useful to list all specific references by the paragraph numbers of the respective sections of each applicable document.

(5) Identification of the organizational units or work areas to be audited and, when appropriate, organizations to be notified.

(6) Identification of the lead auditor and audit team members. (7) Sequence of audit events or references to a separate document containing

the audit schedule. The audit schedule is a part of the formal audit plan documentation as described in Section 3.3.3 and illustrated in the Annex, Examples 6 and 7.

(8) Identification of standard and specifically prepared audit procedures and check-lists to be used.

(9) Identification of the language in which the audit will be conducted, when relevant.

An audit plan should be subject to review and approval. Depending on the situation in which it will be performed, the type of audit and other relevant factors, review and approval may be performed by the management of the auditing organization or a special review board may be formally convened. The situations in which a formal review of the audit plan may be required result: (1 ) when the audit is intended to be a comprehensive evaluation of the overall QA programme; (2) when audit results are to be used to decide whether to accept or reject some crucial portion of the QA programme; or (3) when there is suspicion that serious problems exist in the implementation of the QA programme.

When a decision is taken that a formal review of the audit plan is necessary, an audit plan review board will be appointed. The audit plan will be analysed and reviewed by the board and the review should result in approval of the plan and schedule and authorization for the audit team to proceed.

3.4. Audit team organization and preparation

The management of the auditing organization should select a qualified lead auditor and a suitable number of individual auditors. The assigned team leader is responsible for preparing for the conduct of the audit in accordance with the established individual plan. Persons who have direct responsibilities in the area being audited shall not be involved in the selection of the team or be members of it. To ensure organizational independence of the audit team, the auditors

12

should not report to a supervisor who is directly responsible for activities the auditors must examine.

The number of persons to be assigned to an audit team will depend on the time available to perform the audit, the purpose and locations of the audit and the complexity and specialization of activities to be reviewed. In the determination of the size of the audit team, it should be noted that an internal audit can be covered by a small number of auditors over a longer period of time. An external audit can be more efficient and cost effective when performed by a larger team in a shorter time because of such factors as personnel availability and travel require-ments. However, the number of auditors should be optimized so that the audit activities will be covered over the scheduled time but the auditors do not inter-fere with each other and thus reduce efficiency. In many cases, one well trained and experienced auditor can perform a meaningful and effective audit in a reasonable time.

3.4.1. Personnel requirements

When an audit team is assigned, a qualified and experienced lead auditor shall be designated. Depending on the scope of the audit, members of the team should be selected f rom the QA organization and from among experienced technical specialists. Quality assurance personnel designated to the audit team should be trained and experienced in the auditing techniques and technical specialists should be used when necessary. When formal qualification and certification are required for the lead auditor, a certification programme should be organized by the auditing organization. An example of the format of a lead auditor qualification record is given in the Annex, Example 15, an example of recommended education and experience qualifications for lead auditor is given in Example 16.

Each member of the audit team should be trained in accordance with the requirements of the auditor 's organization. Normally, formal certification is not required but nevertheless an appropriate technical background and education are necessary. In addition to education, experience and training in auditing techniques, a potential auditor should also have attributes related to human relationship and skill in communication.

The auditor 's training should include appropriate orientation in all performance standards and criteria used for QA programme audit. The training should also provide general and specialized instruction in methodology and techniques. General training should include audit objective, organization, preparation, performance and reporting. Specialized training should include methods of examining, questioning and evaluating activities subject to audit and documenting and reporting of findings. Classroom training should be supplemented by on-the-job training so that inexperienced personnel act under direct supervision of an experienced and qualified auditor.

13

3.4.2. Audit team preparation

After assignment of the audit team, enough time should be allowed for the team to prepare. It is the responsibility of the lead auditor to organize and conduct the team's preparation, which should include the following:

(1 ) Familiarization of the audit team with the scope of the audit and the area to be audited. This includes appropriate orientation in the performance standards to be used, quality assurance manuals, procedures, specifications and other relevant documents. In addition, work areas to be audited should be identified with respect to such information as right of access, right to talk with indivudual workers, safety requirements, etc.

(2) Background information about the organization to be audited. This should include information resulting from previous audits such as trouble reports, areas of weakness, etc. The information should also include any opposition that may be anticipated in certain audit areas.

(3) Discussions of the audit plan and schedule including strategy of the audit. (4) Assignment of work to each member of the audit team including the

definition of work area and respective activity. (5) Practical arrangements for the audit team such as travel, accommodation, etc.

On the basis of his assignment in the audit process, each member of the team should review existing and develop supplementary procedures and check-lists when necessary for the scope of the assigned task. These should be reviewed by the lead auditor and discussed with the team.

To prepare satisfactory check-lists, auditors should be allowed sufficient time to review all relevant QA documents of the auditee such as QA manuals, procedures, and specifications and to understand fully what is to be examined or investigated.

The audit team preparation may involve meetings of the team, taking into account all elements of the plan and schedule and the specific task assignment of each member. It is expected that at this stage each member of the team will be fully aware of his duties and responsibilities.

3.5. Audit notification and initial contact with auditee

The auditing organization should not i fy the auditee in advance of a planned audit. The time frame for notification will depend on such things as the type (external or internal) and scope of the audit and other relevant factors. For an internal audit, notification may be relatively short, particularly when an overall schedule has been established. For an external audit, one to three months is considered reasonable notice for a regularly scheduled audit.

Notification includes establishment of purpose, scope and authority. In principle, there is a difference between formal notification and the contacts

14

established between the audit team and the auditee's organization to collect all necessary documentat ion in preparation for the audit. Notification may be given by letter, telephone or personal contact, depending on the relationship between the auditor and the auditee. However, it is expected that all external audits should be preceded by a formal letter of notification transmitted through official channels, which includes at least purpose, scope and schedule. Any additional information deemed appropriate should be also included. For an internal audit, formal notification is not always necessary, but a written notice should be sent to the responsible persons in those areas to be audited.

If the names of the auditors are known, they should be communicated to the auditee together with the notification. Otherwise these names should be transmitted as soon as they are selected. In the case of an audit of a supplier abroad, the language in which the audit is to be conducted should be specified.

Some organizations undertake unannounced audits. In the case of internal unannounced audits, this practice should be established in the organizations' policies and procedures. To conduct unannounced external audits generally requires prior agreement between the parties involved. However, there are strong arguments against unannounced audits: the negative reaction which may be provoked in the auditee organization, the practical problems of contact with auditee personnel, and collection of essential information.

Following the initial notification of the scheduled audit, working arrangements should be established between the auditee and auditing team. The contacts established should be used to obtain the auditee's QA programme manual, organization charts and other relevant documents to allow the auditors to identify the auditee's internal performance standards and procedures applicable to the audit scope. Also through these contacts administrative details such as schedules, assignments and arrangements such as accommodations should be worked out.

Contacts between parties during the preparation phase can be particularly useful when the team is not familiar with the organization to be audited. In these circumstances the lead auditor may carry out a preliminary survey, which allows him to familiarize himself with the auditee's organization, facilities, products, personnel and QA programme documents. With this knowledge, an efficient plan can be developed whereby the team can be optimized and the audit can be carried out in the most economical manner possible.

3.6. Pre-audit conference

At the start of the audit a conference should be conducted involving the audit team and the management and staff of the organization to be audited. The objective of the conference is to develop contacts between parties and to set ground rules, including confirmation of the audit scope and schedule, discussion of the sequence of activities, establishment of channels of communication, and determination of a time for the post-audit conference. When a pre-audit conference is not

15

considered necessary, the team leader should send an agenda of the audit or the complete audit plan to the audit management.

To ensure the success of the pre-audit conference (and indeed of the entire audit process) it is essential to establish a positive psychological environment in dealing with the staff of the audited organization.

The pre-audit conference is conducted by the audit team leader, whose responsibility it is to prepare the conference, to act as chairman and to speak on behalf of the team. To ensure that all key points are covered, the lead auditor should prepare a conference agenda, which may include the following:

(1 ) Introduction of the audit team members and their counterparts in the auditee organization.

(2) Presentations by the team leader and discussions of: (a) scope (b) performance standards (c) methods

(3) Discussion of schedule. (4) Nomination of persons responsible in the auditee's organization for each

area being audited. (5) Establishment of a time for the post-audit conference.

In his presentation the lead auditor should first state the audit objective. (This may be particularly useful in aiding organizations without experience in audits to obtain a clear picture of purpose, scope and methods.) If a previous audit has been performed previously, the relationship between these two audits should be explained, particularly with respect to any corrective action required earlier.

The team leader should also explain which reference documents were used to develop the plan and which performance standards will be used for the audit. The appropriateness and validity of these documents should be confirmed in discussions. Methods to be used should be explained to the auditee. The lead auditor should state which types of documents will be reviewed in both activities and records. He may also present prepared check-lists to clarify the type of information the auditors will be looking for. He should indicate the work areas and activities to be reviewed and identify in discussion those in-progress activities which he will review.

All administrative matters should be clarified in discussions. There should be mutual understanding of the schedule (including all proposed changes and modifications), approximate timing for each portion of the audit, list of persons to be interviewed, etc. The time for the post-audit conference should be identified and a provisional list of the persons expected to attend should be established.

The pre-audit conference should be used by the audit team to form general impressions of the auditee, which will be valuable, for example, in understanding the reactions of the auditee's staff during interviews and in framing the language and thrust of the report.

16

4. AUDIT PERFORMANCE

4.1. Audit techniques

This section offers guidance on the techniques which may be used to ensure the effective conduct and successful outcome of an audit after the preliminary introduction of the pre-audit conference has been completed.

Techniques described include document sampling methods; document reviewing and interviewing; witnessing implementation of work instructions; tracing of processes and systems; and corroboration, such as by performing independent tests, inspections or examinations. It should, however, be clear that the set of techniques described in this section should not be considered as a closed list of tools at the auditor's disposal. The application of such techniques should, by no means, limit the auditor's freedom to act on the basis of his understanding of the situation and his experience to achieve the most reliable results from the audit he is conducting.

4.1.1. Sampling

It is generally impracticable to examine all available documentation during the course of an audit. Hence auditing the effectiveness of implementation of the QA programme usually involves random selection of samples of procedures, instructions and drawings and the records which verify that these have been correctly followed and satisfied.

At the planning stage the auditor should prepare a list of the different types of documents to be examined, e.g. purchase orders, manufacturing drawings, inspection records. For each type the auditor should estimate the amount of relevant data available and establish and record the sample size.

Sample selection can either be done mathematically by statistical sampling methods, or more subjectively by personal or professional judgement on the part of the auditor. The advantages of statistical sampling are the determined random-ness in the selection of samples and the mathematical probability of conclusions drawn from the sampling process; the inability of a qualified auditor to exercise objective judgement is a definite disadvantage. It is acknowledged, however, that a good auditor can detect significant problems which are not discovered by mathematical methods, possibly by bringing together clues apparent from auditing related areas or established by tracing sequential activities. In cases where an auditor uses his professional judgement, an examination of the auditee's previous performance may be of substantial value in guiding the selection of the most appropriate sample of documents.

17

4.1.2. Reviewing documents

It is essential to review the auditee's QA manual and associated implementing procedures for compliance with the specified QA and technical requirements. In addition to detecting deficiencies in the programme, in reviewing the documents, the auditor obtains valuable initial information regarding the auditee's operations. The review should be carried out during the pre-audit phase wherever possible to assist in the development of a specific check-list. Such reviews do not necessarily provide conclusive evidence of a non-compliance with specified requirements and in many cases further evidence must be gathered f rom other documentat ion (e.g. records of ou tpu t of the procedure, work instructions, and drawings or by interviews with the relevant auditee personnel). However, time spent reviewing documentat ion before an audit can considerably reduce the amount of reading, studying and analysing which would otherwise be necessary during the audit.

Reviews of the output of programmatic procedures, e.g. purchase orders, process control instructions and drawings, provide evidence of satisfactory implementation of the auditee's programme. This is not always conclusive, however, and further necessary evidence can normally be accumulated by inter-viewing and observing working practices. Check-lists, such as those discussed in Section 3.3.5, are usually constructed so that at the appropriate time the auditor can review the compliance of the manual and procedures with specified require-ments and compliance by the auditee with his procedures.

4.1.3. Interviewing

Although some audit objectives can be achieved by review of documented objective evidence alone, such as record validation, certain requirements may be audited only through interviews. Interviewing provides important audit information particularly where the responsibilities and authori ty of personnel are being audited; e.g. in some cases there is no generally available evidence of implementation of documented responsibilities. Interviewing supervisors and personnel at working stations can provide valuable insight into their knowledge and understanding of the documented practices and hence the likelihood of their compliance with appropriate performance standards.

The type of questions which may be asked during an interview should be concerned with the why, when, where, who, which and how. Examples of questions are as follows:

Why does only the purchasing group review tenders? When does the QA group review customer requirements? Where are QA records stored? Who will conduct and witness tests? Which processes are considered special? How are inspection personnel trained?

18

It is essential for the auditor to establish an adequate environment for an interview, e.g. free f rom frequent interruption. He must always strive to be fair, for instance by suggesting improvements but not formally reporting minor conditions and by endeavouring to demonstrate that he is not conducting a 'witch-hunt ' . The auditor should also show understanding of any explanations given for problems revealed by the audit. The auditee will more readily accept the need for change when he recognizes that the auditor understands the position but is nevertheless firm in his convictions.

4.1.4. Witnessing implementation

The most direct technique for establishing compliance with auditee procedures and work instructions is to witness implementation of the activities at the work-place. However, witnessing implementation in this context should not be used for item acceptance. This process can be used effectively, for example, in areas where special manufacturing and inspection processes are taking place. Typical examples include observing welding operations, heat treatment, and destructive and non-destructive testing and auditing compliance with the relevant procedures, instructions and drawings.

If this technique is used, the procedures and systems relevant to the activity should be reviewed before the audit wherever possible and any difficulties in understanding clarified with the auditee. Specific explanations of the procedure which may be needed are best accomplished at the workplace, where verification of compliance of the operation with the procedures can be carried out simultaneously.

4.1.5. Tracing

An effective method of verifying that a process is being carried out correctly is to select a specific transaction and then trace the processes that have been accomplished to arrive at it. Examples of specific transactions are a specific non-conformance report or a process being carried out on a particular product . Assuming that the transaction selected for tracing is a valid sample, this technique can provide valuable evidence about whether the relevant processes performed have been accomplished in accordance with the procedures.

The general approach to planning an audit using this technique is as follows: (1) decide the product line, document type or service and the stage in the process to be used as the starting point ; (2) identify the major steps that would typically be expected to have been followed to arrive at the starting point; (3) identify the key features of each of the documents controlling the major steps in a check-list or audit procedure; and(4) decide on a sample size.

Samples should be selected on the basis of reproducibility of the process, e.g. f rom a standard product line. Where processes are likely to be different because of variations in complexity or other unique features, at least one sample

19

of each product line, service or document should be selected. More samples may be taken if the audit objective is to ' trouble-shoot ' a particular activity; a sample size of three is usually considered satisfactory in such cases.

A sample check-list used for auditing with the tracing technique is given in the Annex, Example 17.

4.1.6. Corroboration

Corroboration is a technique by which a particular process is repeated independently of those who originally performed the process. It is rarely used as an auditing technique because of the cost and time involved. Furthermore, corroboration in this context should not be confused with item acceptance.

Corroboration can serve as an effective way to verify that a particular process was performed correctly, but its value as an audit technique rests on the assumption that the item or activity selected for corroboration is a representative one. Examples of corroboration include independent tests, inspections or examinations of a particular item using the same procedures and equipment as the original tests, etc.

Reinspection of items involves removing particular parts or assemblies and performing a sample inspection of selected characteristics. The characteristics to be inspected must be carefully considered: tight tolerances and interface dimensions, electrical functions and variables data are aspects which should be verified. A sample inspection of minor characteristics provides a measure of auditee compliance with instructions, drawings, etc.

A sample data sheet check-list for corroboration is given in the Annex, Example 18.

4.2. Recording audit findings

In performing the audit, the auditor will be determining findings and obser-vations. A finding is a statement of fact regarding non-compliance with established requirements. An observation is an opinion regarding an unsatisfactory condition not covered by a specific requirement, or a procedure, practice or instruction whose effectiveness could be improved.

An auditor must provide sufficient objective evidence to support his findings and observations and provide detailed information to the auditee to allow fur ther investigation for corrective action. The auditor should not , therefore, rely on memory but must record evidence as he proceeds.

Thus the items examined in the determination of compliance or non-compliance should be uniquely identified and recorded. If the check-list explicitly identifies the items to be examined this may provide sufficient evidence. In any case, during the audit the auditor should use a satisfactory recording system (e.g. forms for the purpose or at least notebooks) to enable easy retrieval at any

2 0

later date relevant to the required retention period for the data. This information provides the basis for the formal recording of the findings and observations to be included in the audit report. Where evidence of examination of documentation is to be recorded, it is essential to note revisions document numbers, titles and revisions. It is also important for evidence to be recorded as soon as it is seen, i.e. before items are returned to files or storage. A sample findings and observations recording form is given in the Annex, Example 19.

It is good practice to reveal findings and observations to the individual assigned to accompany the auditor so that he understands the deficiency. Further-more, if a responsible representative of the auditee can verify the recorded facts this is also advantageous. A method of recording the facts of a finding is shown in the Annex, Example 20.

4.3. Audit team co-ordination

Auditing requires close co-ordination among team members. The process of co-ordination generally starts with the appointment of a team leader, who normally determines the assignment of team responsibilities, briefs each member at the planning stage, and is subsequently responsible for ensuring that every team member is able to (and does) perform his assigned duties.

During the course of the audit there should be frequent exchange of infor-mation between members of the team, which will allow the team leader to make necessary changes in direction and assignments to improve the evaluation of a situation and achieve the objectives defined in the audit plan. These exchanges also allow each member to relay information which may be pertinent to another part of the audit.

A meeting of the audit team should take place before the post-audit conference. At this meeting the team leader should co-ordinate (1) discussion of findings and observations, (2) evaluation of findings and observations to identify any which can be consolidated as generic problems, and (3) presentation of findings and observations to ensure that these are factual and can be substantiated.

5. POST-AUDIT ACTIVITIES

The audit performance can be considered complete when all the facts related to the scope of the audit have been identified, objective evidence has been examined to determine compliance with evaluation criteria, and all findings and observations have been recorded and documented to the extent necessary for preparation of the audit report.

Post-audit activities then consist of the following phases: (1) preparation and performance of the post-audit conference with the management of the audited organization, (2) preparation and submission of the audit report, including

21

suggestions for corrective actions where appropriate, and (3) follow-up activities, including re-auditing when necessary. Each of these includes several independent activities which are the responsibility either of the audit team or of the audited organization.

5.1. Post-audit conference

To conclude the audit, a conference is organized between the audit team and the representatives of the audited organization. All members of the audit team, a representative of management from the audited organization, and other persons delegated by the auditee management should attend. The primary objective of the post-audit conference is to provide an opportunity for the team to present its findings and observations to the management and to ensure that the audited organization understands them.

To meet the objectives of the conference the lead auditor and the team members should ensure that the conference is well prepared, which can be done during a team meeting in which a careful evaluation and analysis of findings and observations is made. Each auditor should review and consolidate his findings to ensure that his meaning is clear and his terminology is correct. The lead auditor should review all findings and observations to determine whether generic problems are identifiable or whether a finding is corroborated by others so that consolidated findings could be reported. Suggestions should also be discussed.

The agenda of the post-audit conference should include a summary assess-ment of the audited QA programme based on the results of the audit (presented by lead auditor), a presentation of findings, a presentation of observations, and a discussion of suggestions for corrective action, where appropriate.

The lead auditor should chair the post-audit conference. He is also responsible for presenting a summary of the audit and suggestions for corrective actions. The main part of the conference should consist of presentation and, if necessary, an explanation of the findings. In the presentation the lead auditor may choose to present all findings himself or to have each member present his own. Regardless of the method of presentation, it should be systematic and satisfy the following practices:

(1) All findings concerning a single requirement or, in the case of an audit of broader scope, findings concerning a single control should be grouped together.

(2) In presenting findings, the following items of information are indispensable:

(a) What was the requirement? (b) What was the deficiency? (c) Where was the deficiency observed (organizational unit, physical

location)?

22

Occasionally a finding is determined to be invalid or is worded in such a way as to imply something different than the auditor intended. The post-audit con-ference gives an opportunity to clarify all misunderstandings, although it is preferable that questions of fact be resolved before the final conference.

Related observations should be grouped together because although they may be insignificant individually, when grouped together they can indicate a possible trend that deserves mention. The lead auditor should also summarize, but not detail, exceptionally good practices observed during audit.

The post-audit conference should also consider the question of corrective action. For many findings, it will be agreed that a non-compliance does exist. In such cases the audited organization may be able to define a corrective action itself and establish an approximate schedule for its accomplishment during the post-audit conference. In some cases, however, the audited organization will require time to investigate further a finding and establish suitable proposals for corrective action. It is important that the team leader recognize that the responsi-bility for defining corrective action lies with the audited organization and not with the audit team. If followed by the audited organization, formal suggestions by the audit team for specific corrective action would prejudice the future independence of the auditing organization.

5.2. Reporting

The audit report is the final product of the auditing team, and its content should be considered during the audits and at the post-audit conference. Findings and observations in draft form as well as all remarks and notes taken during the audit should be used as aids for report preparation; the main elements of the report are prepared before and presented during the conference. Discussion during the conference should clarify all unresolved questions and verify all assumptions which the audit team may use in formulating its findings and conclusions.

As the final and complete documentation of the audit, the format of the audit report should fulfil the purpose of the audit. To satisfy this goal a report should include the following main items: purpose and scope of the audit, summary of results, findings, observations, suggestions for correcting non-conformances or deficiencies as appropriate, and response requirements.

The format of the report may vary from a standardized form which should be filled in by the auditor (see Annex, Example 21) to a free-standing text. The audit report should be prepared and submitted to the auditee as soon as possible (since it represents a requirement for action to be taken by audited organization it should be issued within a reasonable time, taking into account the urgency of the corrective measures). It is, however, general practice for the audit team to return to its home organization to prepare the report.

The audit report should identify the audit to which the report is related. This is normally done by appropriate use of headings and titles such that the

23

report can be identified and readily retrieved when used as a QA record. It is also important that the heading provide a means for identifying the audited organization or project.

The purpose and scope of the audit should be stated at the beginning of the report. The statement of purpose should define the reason for the audit, and the statement of scope should clearly define which area was covered and which activities or work areas were audited. It is useful to identify all activities that were audited regardless of whether or not findings are reported.

The summary of results should contain the auditors' evaluation of the QA programme's adequency and effectiveness. This should be an overall assessment without entering into details. A summary statement should be included regarding any exceptionally good aspects of the programme which were observed. Recorded findings should merely be referenced in support of the impression the auditor obtained.

The summary should be followed by detailed findings and observations, including all identified non-compliances, deficiencies or other conditions adverse to quality. The wording of each finding should be such as to clearly identify the deficiency so that it can be used as the formal foundation for corrective action and determination of the effectiveness of that action. When each finding is recorded separately (see Annex, Example 21) the report should list the findings and refer to the attached, completed findings forms. The report of a finding should consist of a concise statement of generic problems that will result in an investigation into the underlying causes and corrective action required to prevent recurrence. The properly written finding should include a precise statement of what the requirements were, a description of the way the audited activity or item failed to comply with the requirement, and evidence supporting each finding. For each finding and observation presented in the report a response is required from the audited organization.

It may be the practice of the auditing organization to make suggestions for correcting non-conformances or deficiencies. However, there is some risk of loss of independence by the audit team and/or the auditing organization attached to proceeding this way. In any event, the making of such suggestions shall not reduce the responsibility of the audited organization for determining corrective actions.

The audit report should be transmitted to the management of the audited organization, accompanied by a short summary of the general results and including a date for a response to the report findings. The Annex provides one example of an audit report including transmittal letter (Annex, Example 22) and another which includes transmittal letter, findings in a standard format and response by the audited organization (Example 23).

5.3. Response

The management of the audited organization should acknowledge receipt of the report, confirm its agreement and commitment, and review and investigate

2 4

all findings and observations to determine the appropriate corrective action. The response, which may use the auditors' audit report format (Annex, Example 23), should clearly state the corrective action taken or planned. The response should also define any remedial action taken with respect to any non-conformance identified in the report. It should also specify the scheduled completion date for the corrective action. If the auditee management does not agree with the finding or considers a corrective action inappropriate, the response should specify the reason for no action and explain how the reported deficiency can be accepted without any adverse effect on quality. The response is a commitment of the audited organization to corrective action. It should be considered as such by the management of the auditing organization and the audit team leader.

5.4. Follow-up

When necessary the auditing organization should take follow-up actions as follows: (1) obtain the written response to the audit report; (2) after obtaining the written response, evaluate the response, including the adequacy and effective-ness of the proposed corrective action; (3) confirm that corrective actions are accomplished as proposed and scheduled; and (4) determine that the action taken was effective.

Confirmation of the timely completion of corrective action may be accomplished through written communication, through a follow-up audit, or through other appropriate means. Determination of the effects of corrective action may involve a considerable lapse of time. It may well take the time up to the next scheduled audit, when the audit plan should also include a check for the conditions of the original finding. If the next audit does not record the same (or similar) non-conformance the corrective action can be judged as having been effective.

6. RECORDS

The following type of audit documents should be considered as QA records and be retained and maintained according to the requirements and recommen-dations of the Safety Guide on Quality Assurance Records System for Nuclear Power Plants (IAEA Safety Series No.50-SG-QA2): audit schedules, audit plans, audit reports, completed audit check-lists, auditor qualification records and audit follow-up records (re-audit reports, etc.). These records should be maintained for evidence of evaluation of the QA programme and can be used as evidence of the adequacy and effectiveness of a QA programme in the hierarchical chain of nuclear power project participants. However, in accordance with the Safety Guide on Quality Assurance Records System for Nuclear Power Plants, they shall be designated 'non-permanent' records.

25

ANNEX

EXAMPLES OF AUDIT DOCUMENTS

27

Example 1. Extract from a quality assurance programme description

Proc. No. QP 18.1

Rev. 4

Date October 31, 1980

Page 1 of IV

AUDITS

I. GENERAL

Internal audits of the quality assurance programme shall be planned and executed to assure that the programme complies with the requirements of the Code of Practice on Quality Assurance, that it is effective and that it is correctly executed and well maintained.

The audits shall cover all activities contained in the quality assurance programme.

II. FUNCTION

The responsible management shall review the results of the audit and shall take action to correct possible deficiencies revealed by the audit.

The quality assurance programme may be subject to audits periodically or on a random basis, or both.

The audit shall be performed in accordance with written procedures and instructions by qualified personnel having no direct responsibility in the areas being audited.

TYPICAL COMPANY LIMITED

QUALITY ASSURANCE MANUAL

CONDUCT OF QUALITY ASSURANCE DEPARTMENT

QUALITY AUDITS

2 8

Example 2: Conduct of quality assurance department audits

TYPICAL COMPANY LIMITED

QUALITY ASSURANCE MANUAL Proc. No. QP 18.1

Rev. 4 CONDUCT OF

QUALITY ASSURANCE DEPARTMENT QUALITY AUDITS

Date October 31, 1980

Page 1 of 8

1.0 APPROVAL:

Manager of Quality Assurance

2.0 PURPOSE:

To delineate responsibilities and requirements for the preparation, performance and reporting of Quality Assurance Department Quality Audits.

3.0 SCOPE:

This Quality Procedure applies to all management and activity audits performed by the Quality Assurance Department.

4.0 RESPONSIBILITIES:

4.1. The Manager of Quality Assurance is responsible for implementation of this procedure.

4.2. The cognizant Assistant Manager of Quality Assurance is responsible for the following actions:

1. Scheduling of audits in accordance with Quality Procedure 18.2. Scheduling includes the selection of a Principal Auditor, qualified in accordance with Quality Procedure 2.9, for each audit. Selection may include accompanying auditors. Audits shall be performed under the direction of a qualified Principal Auditor.

2. Ensure audits are conducted in accordance with this procedure. 3. Review of each audit report for accuracy, completeness, proper format and

distribution.

4.3. The person designated Principal Auditor for each audit shall assure that the audit is conducted in accordance with this procedure. The Principal Auditor shall cause a written report of resultant findings to be prepared. He shall ensure completion of corrective action for those audit findings that identify deficient conditions.

2 9

TYPICAL COMPANY LIMITED

QUALITY ASSURANCE MANUAL Proc. No. QP 18.1

Rev. 4 TYPICAL COMPANY LIMITED CONDUCT OF

QUALITY ASSURANCE DEPARTMENT QUALITY AUDITS

Proc. No. QP 18.1

Rev. 4 TYPICAL COMPANY LIMITED CONDUCT OF

QUALITY ASSURANCE DEPARTMENT QUALITY AUDITS

Date October 31, 1980

Page 2 of 8

TYPICAL COMPANY LIMITED CONDUCT OF

QUALITY ASSURANCE DEPARTMENT QUALITY AUDITS

5.0 PROCEDURE:

The purpose of the audit programme is to verify the development and implementation of the Quality Assurance Programme. Management audits are performed to determine if the QA programme has been developed and implemented. Activity audits are supplementary to management audits and are performed to verify implementation of a specific function of the QA programme. Activity audits may be planned so that a series of such audits will verify the development and implementation of a portion of the Quality Assurance Programme.

5.1. Preparation:

1. The overall programme planning and scheduling of audits is in accordance with QP 18.2, "Scheduling of Quality Assurance Department Audit Activities".

2. The planning and preparation for the content of each audit shall consist of the following:

A. B. C. D. E.

F. G.

H.

I.

J.

3. Audit Plan:

An individual audit plan which, as a minimum, contains the information stated in 5.1.2.1 through 5.1.2.6 shall be developed for each audit. The individual audit plan for management audits shall be approved by the cognizant Assistant Manager. Activity audit plans shall be approved by the cognizant Assistant Manager or his on-site designee. The audit plan shall be provided to each auditor.

3 0

Identification of audit scope (subject). Identification of activities to be audited. Identification of organization or person to be notified. An audit agenda. Identification of applicable procedures, standards, instructions, codes and regulatory requirements. Identification of audit check-list or procedures. Review of pertinent policies, procedures, standards, instructions, codes, and regulatory requirements (Audit Criteria). Review of previous audit reports. Specific attention shall be given to findings of deficient conditions or areas of weakness in previous audits. In this manner, identification of new subject areas and a check of implementation of formerly resolved findings can be provided for. Review of organization (internal and external) and contractual relationships and responsibilities of the organization to be audited. An understanding of these relationships and responsibilities is essential in measuring QA Programme effectiveness. Selective reviews of quality-related procedures used by the audited organizatior shall be performed. These reviews will be against applicable regulatory and QA Manual requirements.

TYPICAL COMPANY LIMITED

QUALITY ASSURANCE MANUAL Proc. No. QP 18.2

Rev. 7 CONDUCT OF

QUALITY ASSURANCE DEPARTMENT QUALITY AUDITS

Date October 31, 1980

Page 3 of 8

4. Audit Check-lists or Procedures:

The audit shall be conducted using an approved audit check-list or procedure and appropriate portions provided to each auditor. The check-list or procedure shall identify criteria and reference applicable documentation from which specific criteria are drawn. The check-list or procedure shall serve as a guide to ensure adequate coverage of audited activities. It should ensure depth and continuity of the audit. Use of a previously approved check-list or audit procedure is permissible if suitable to the audit to be conducted. Previously approved check-lists and procedures will be reviewed to ensure they reflect any changes to criteria or referenced requirements since being approved. This review shall be certified on the check-list or procedure by the Principal Auditor. Audit check-lists shall be pre-pared in a format similar to Form QP 18.1—3. Audit procedures shall be approved prior to use by the cognizant group Assistant Manager or Manager of Quality Assurance. Audit check-lists shall be approved prior to use by the cognizant group Assistant Manager or by the Principal Auditor or On-Site Supervisor, if delegated in writing by the Assistant Manager.

5. Specialists and Consultants:

The cognizant Assistant QA Manager should request the services of a specialist or consultant to assist in performing an audit in an area where additional QA expertise is required. Specialists or consultants shall not have any direct responsi-bilities for the areas which they audit.

6. Notification:

Notification assures the presence of appropriate personnel from the audited organization and permits identification of major conflicts, due to unforeseen events or activities, which might preclude an efficient audit. In the case of such conflicts, the audit can be rescheduled. Organizations to be audited shall be notified, as a minimum, several days (e.g. 4 working days) prior to the scheduled audit commencement date. Notification normally consists of a telephone call for internal audits and a letter for external audits. Unannounced audits may be performed with prior agreement of parties involved. The notification requirement may be waived for on-site activity audits conducted by the construction and operating plants' groups.

5.2. Performance:

1. Pre-audit Conference:

An informal pre-audit conference shall be conducted with management of the organization to be audited, except it may be omitted for on-site activity audits

31

TYPICAL COMPANY LIMITED

QUALITY ASSURANCE MANUAL Proc. No. QP 18.2

Rev. 7 CONDUCT OF

QUALITY ASSURANCE DEPARTMENT QUALITY AUDITS

Date October 31, 1980

Page 4 of 8

conducted by the construction and operating plants' groups. The Principal Auditor shall use this conference to: Present the audit plan, introduce auditors, meet counterparts, confirm the status of the audited activity, establish audit scope, discuss auditing sequence, establish communication channels, and plan the post-audit conference (exit interview).

2. Examination:

The following guidelines, as applicable, shall be adhered to by the auditors during the examination phase of the audit:

A. Audit team activities should be co-ordinated with the audited organization throughout the audit, so that the audit findings and circumstances surrounding the observation will be completely understood.

B. Auditors shall be alert for unplanned, unusual or unexpected circumstances which may warrant further investigation. Circumstances requiring immediate attention shall be reported promptly to the responsible supervisor/manager. Immediate corrective action in accordance with QP 16.1 shall be initiated, if applicable.

C. Properly approved check-lists or procedures established per Section 5.1.4 shall be used.

D. Objective evidence shall be examined to determine compliance with Quality Assurance Programme requirements.

E. Each element selected for examination shall be audited in sufficient depth to determine whether or not satisfactory compliance has been achieved.

F. When non-conforming areas or practices are identified, applicable procedures and documentation should be reviewed for adequacy.

G. When procedures are not available or are inadequate, further investigation shall be conducted to determine if additional