APTA SAFETY MANAGEMENT PROGRAM DEVELOPMENT AND

2008

ISSUE 023 – Internal Safety Audit

Dated 1/15/04

REV: 5/22/2008

Application: Bus, Rail Transit, Commuter

Rail Safety Management Programs

References:

APTA SAFETY MANAGEMENT PROGRAM DEVELOPMENT AND IMPLEMENTATION GUIDELINE

EXPECTATIONS This guideline is provided for properties to use with any of the APTA Safety Management Programs to assist them in the development of higher quality and continuous improvement process in their safety and security applications. Program members should be able to use this information to build effective Corrective Action Plans to manage and mitigate risks associated with identified hazards as low as reasonably practicable.

Effective: January 2004 DOLR: May 2008

APTA DEVELOPMENT &

IMPLEMENTATION GUIDELINE

FOR

Internal Safety Audits

* * *

Provided to support the

Safety & Security Management Programs

AMERICAN PUBLIC TRANSPORTATION ASSOCIATION

* * * * * * *

William P. Grizard, Director- Safety

William W. Millar, President AMERICAN PUBLIC TRANSPORTATION ASSOCIATION

1666 K Street, NW Washington, DC 20006


Table of Contents

Chapter 1 - System Safety Program Plan Requirements for Internal Audits Introduction Overview Program Functions

Chapter 2 – Internal Audit Terminology Definitions

Chapter 3 – Internal Audit Planning Phase Development Scope Notification Requirements Scheduling Staffing Audit Elements

Chapter 4 – Internal Audit Preparation Notification Audit Outline Audit Checklist Special Checklist Reference Criteria Types of Verification Documents Data and Information Verification

Chapter 5 – Internal Audit Execution Audit Process Overview Audit Techniques Pre-Audit Conference Document Review Data Taking Audit Review Post-Audit Conference Identification of contested items


Chapter 6 – Internal Audit Result Analysis Audit Findings Qualifying Evidence Quantifying Results Identification of Non-conformance Hazard analysis of Non-conforming Items Recommendations for Corrective Action Prioritizing Corrective Actions Trend Analysis

Chapter 7 – Internal Audit Evaluation Review Audit Goals and Objectives Correspondence with SSPP Conditions for Re-auditing

Chapter 8 – Internal Audit Reporting Report Process Overview Preliminary Report

Transmittal of Preliminary Report Response to Report Final Audit Report Overview Audit Summary Audit Findings Corrective Action Transmittal of Final Report

Chapter 9 – Internal Audit Tracking Scope Responsibilities Status Reports Tracking System Schedule Adherence Audit Records

Appendix


PREFACE This guideline on Internal Safety Auditing has been developed to form a better understanding of the methodology and techniques involved in the planning, conducting, evaluation, and reporting of an effective audit and its relationship with the Agency's system safety program plan. The methodology and techniques described herein are not intended to be the only valid methods for establishing and implementing an internal safety audit program. Rather, they are intended to develop a consistent method for conducting internal safety audits as described within the American Public Transportation Association (APTA) Safety Management Program Manuals for the various modes (Commuter Rail, Bus and Rail).

Public Document Disclosures Each agency performing internal audits needs to verify applicable state and federal legislation on the public disclosure of the documentation generated as the result of fulfilling internal auditing requirements. The Internal Audit program may be considered an open process subject to public disclosure. All internal audit reports produced may be considered public record. Background information, checklists, notes, non compliant items, corrective action plans, status reports and other documentation prepared as part of the audit process may or may not be protected from public disclosure. [NOTE: Due to the sensitive nature of security program elements, audit information and all background information, checklists, notes, and other related audit documentation may be protected under the provisions of CFR Title 49, Part 659, Subpart B, Section 659.21 (b), depending upon individual state oversight agency standards.]


Chapter 1 - System Safety Program Plan

Requirements for Internal Audits

Introduction The Federal Transit Administration (FTA) issued rulemaking in Title 49 CFR Part 659 which established Rail Fixed Guideway Systems State Safety Oversight. That requirement is adopted by the regulations published by State Oversight Agencies as part of their Program Standard. These regulations include a requirement for the development and implementation of an Internal Audit program within an agency’s System Safety Program Plan. The Bus and Commuter Rail Modes are not currently under specific legal requirements to perform internal audits. However, they are considered essential elements of a proper System Safety Program. APTA has developed Guidelines for System Safety Program manuals for use by Commuter Rail and Bus systems that include internal auditing as one of the major program elements. The manual does not prescribe an absolute format for System Safety Program Plans or for Internal Audit Programs. Instead, it offers a suggested format, along with the type of methodology that will accomplish the purposes of System Safety. This development and implementation guideline is designed to satisfy this same goal as one single program cannot be applied to all agencies in the same manner to be successful. Agencies are encouraged to use this material to fashion their own programs which will be unique to their own organizational context and operating environment.

Overview of an Agency Internal Audit Program The System Safety Program Plan (SSPP) is a top level governing document that establishes the Agency's safety philosophy and provides the means for implementing system safety programs throughout the operational life cycle of the Transit system. In compliance with the federal and state requirements shown above, an Internal Audit program is required as part of the SSPP. The SSPP should provide for conducting internal audits and describe the methodology used for each phase of the Internal Audit program.


Program Functions Implementation of these requirements is usually guided by an SOP on Internal Safety Audit Procedures. The SOP defines the duties and responsibilities necessary for meeting the regulatory requirements by identifying program functions and assigning personnel to accomplish them. The following list of program functions is essential to the proper execution of an Internal Audit program:

Establish internal audit responsibilities for conducting, documenting, correcting and reporting on safety audit activities within the Agency

Verify safety programs have been developed and implemented in accordance with the Agency’s System Safety and Security Program Plan requirements

Assess effectiveness of the Agency’s safety and security effort

Identify program deficiencies

Identify potential hazards in the operational system and weaknesses in the management of system safety programs

Recommend improvement

Verify corrective actions are being tracked for closure

Provide management with assessment of status and the adequacy of the system Safety and security program

Evaluate safety and security programs continuously and comprehensively over a 3 year cycle

The internal safety audit is a compliance type audit whereby information is gathered and facts, including statistical data, are reviewed to verify compliance of safety elements with the System Safety Program Plan and regulatory requirements contained in applicable codes, standards, or regulations. However, the internal safety audit involves more than a compliance evaluation. It also includes an assessment of:

Management participation & involvement

Interdepartmental coordination by management

Managers' awareness and response to potential hazards. The internal safety audit report of findings & recommendations also takes into consideration management effectiveness in implementing the System Safety Program Plan. A sample Internal Audit SOP format is included with this Guideline as an aid to agencies in the development of their own.


CHAPTER 2 – Internal Audit Terminology

Definitions

Audit Systematic, independent and documented process for obtaining evidence and evaluating it objectively to determine the extent to which audit criteria are fulfilled. [NOTE: Current 659 language cites “safety and security reviews,” not “audits.” Additionally, such reviews are to be conducted according to the Program Standard that is created by each State Safety Oversight Agency.]

Auditor Person qualified and competent to conduct audits.

Audit Criteria

Set of policies, procedures or requirements against which collected audit evidence is compared.

Audit Program Sets of audits to be carried out over a planned 3-year cyclical time frame utilizing the APTA Safety Management Program Guidelines and those adopted by the Federal Transit Administration in CFR 49, Part 659.

Audit Report The report generated for each Internal Safety Audit after all events of an audit have been completed. It may include such information as all non conformance items found, recommendations made, corrective actions taken, exceptions noted, contested items, and the Audit Summary.

Audit Summary A narrative description of the state of implementation of the System Safety Program Plan as determined in a specific audit area. The description is based on facts and information compiled during the Audit.

Audit Team One or more auditors conducting an audit, one of who is appointed the team leader.


Certificate of Conformance

A written statement signed by a certified party certifying that items, processes, or systems comply with specific requirements.

Compliance Fulfillment of a legal requirement.

Conformance Fulfillment of a requirement. Specifically, an item on an audit checklist which is found to be in fulfillment of an audit element requirement or the audit criteria.

Corrective Action

Action taken to eliminate the cause of a detected non-conformance or other undesirable situation.

Corrective Action Plan A Plan prepared by the audited department that describes the results of their investigation, actions to take for implementing changes, a schedule for completing each change, and person responsible for completing the changes, and the method chosen to ensure that corrective actions effectively resolve the audit finding.

Corrective Action Program The management process of identification, investigation, and remedial action of non-conformities to preclude recurrence of a condition adverse to the attainment of safety or security policies and objectives.

Discrepancy Non-fulfillment of a requirement. Specifically, an item on an audit checklist which is not in compliance / conformance with the provisions of an audit element requirement or the audit criteria.

Evidence Records, verified statements of fact or other information relevant to the audit.

Exception

An item found during the course of an audit for which there is no criteria established to evaluate conformance, falls outside of the audit checklist scope, or cannot be


statistically validated.

Finding Results of the evaluation of the collected audit evidence against audit criteria. Specifically, a non-conformance item on an audit checklist which, after analysis, indicates a condition adverse to the attainment of system safety or security policy and program.

Inspection The process of measuring, examining, testing, gauging, observing or otherwise comparing an item with the applicable requirements for the purpose of evaluating its conformance.

Internal Safety Audit Plan A documented activity performed in accordance with written checklists to verify, by examination and evaluation of objective evidence, that applicable elements of the System Safety Program Plan have been developed, documented and effectively implemented in accordance with specified requirements and criteria.

Measurement Set of operations having the objective of determining the value of quantity.

Non-conformance Does not fulfill a requirement.

Objective Evidence Data supporting the existence or verity of something. Specifically, any verifiable statement of fact, information, or record, either quantitative or qualitative, pertaining to the value of quantity or quality of an item, process or system.

Procedure

Specified way to carry out an activity or a process.

Requirement

Need or expectation that is stated, customarily implied, or obligatory.


Root Cause Fundamental cause that results in a nonconformance.

Specification

Document stating requirements.

System Safety The application of operating, technical, and management techniques and principles to the safety aspects of a system throughout its life cycle to reduce hazards to the lowest practical level within the constraints of time, cost and operational effectiveness.

System Safety Management An element of management that defines the System Safety requirements and ensures the planning, implementation, and accomplishment of System Safety tasks and activities throughout the system lifecycle.

Test

Technical operation that consists of the determination of one or more characteristics of a given item, process, or service according to a specified procedure.

Validation

Confirmation and provision of objective evidence that the requirements for a specific intended use or application have been fulfilled.

Value of Quality

Ability of a set of inherent characteristics of an item, process or system to fulfill requirements.

Value of Quantity

Measurement of a set of characteristics of an item, process or system to fulfill requirements.

Verification

Confirmation and provision of objective evidence that specified requirements have been fulfilled.


Chapter 3 – Internal Audit Planning Phase

Development

The responsibility for developing an audit plan for each audit conducted must be assigned to a specific person in the agency. The audit plan identifies the intended audit areas, the type of audit, the audit team (where one is needed), and the resources and time expected to conduct and complete the audit activities. The audit plan is a written document that forms the basis for the scope and schedule for the audit. It provides a format describing the requirements for conducting the audit. In developing an audit plan, the following is considered:

Nature of responsibilities and activity of department to be audited

Previous audit findings and accident investigations conducted

Management responsibilities and organizational structure

Audit staff skills & expertise.

Scope In the development of the audit plan, the team leader must consider the following:

Safety & security elements to be audited

Scheduled dates of audit, conferences, reports

Name of auditors, technical specialists, and observers

Audit identification number

Organizations, staff, and activities to be audited

Departments to be audited

Reference criteria

Scheduled audit dates

Identify special conditions relevant to the audit

Identification of verification methods to be used

Notification Requirements The team leader, with concurrence by the department manager to be audited, prepares a preliminary audit schedule as part of the audit plan. This may require scheduling audits from six to twelve months in advance, depending on departmental size and scope of the planned audit.


Scheduling Developing the audit schedule requires a joint effort between the team leader and the department manager(s). The audit schedule describes the activities to be performed during the audit. A review of the schedule is conducted with the affected department manager (s) as to the dates, audit staff and scope of the audit. In establishing an overall audit schedule, the following issues are considered:

Audit scope

Location where the audit is to be conducted

Complexity of audits

Frequency and dates of audit

Reference criteria

Requirements for audit staff

Department personnel requirements

Departments’ schedule

Department notification.

Audit Staff Audits may be conducted by one person or by a team. The audit does not necessarily need to be conducted by safety staff. The audit team may be composed of personnel from within or outside of the agency as long as they are trained and experienced in the technique of auditing and have an appreciation for the area being audited. In order to avoid any potential for conflict of interest, auditors selected must be independent from first line of supervision responsible for the activity being audited. If technical expertise is needed, the team leader will determine the types of technical requirements needed to assist in the audit activities. The size of the audit staff is a usually a function of:

Auditors skills/experience

Scope of the audit

Length of the planned audit

Type of audit performed


Safety Elements to be Audited The audit plan should specify the scope of the audit and which safety elements or specific safety issues will be subject of the audit. Safety elements are identified or prescribed by the system safety program plan, procedures, codes, FTA, State Oversight Agency regulations or best industry practices. For a Rail system, the audit elements must, as a minimum, include 21 [659.19 (a) – (u)] system safety program elements. These elements are required by FTA regulations. However, the Rail systems are not limited to those audit areas. The safety elements for Bus and Rail systems are listed in the Audit Elements Master Index which groups them by audit areas. Each audit element has criteria associated with it, which the team leader uses for establishing the audit reference criteria.

Audit Criteria The audit criteria are derived from the standards, rules, requirements, etc., on which a decision on the scope of an audit is based. Both the Bus and Rail systems have audit areas defined in the Audit Elements Master Index which can used as a reference document in developing the Audit Plan, as well as the System Safety Program Plan, previous Audit Checklists and other types of reference criteria. The integrity of the audit process is often based upon assuring that the appropriate criteria for performing the internal safety audit has been specified. As such, an ongoing assessment during the planning, conduct and release of the results of the audit should evaluate:

Constraints governing the criteria for the issue being audited

Requirements or additional criteria that had not been specified in the checklist or reference documentation

Criteria changes identified during the conduct of the audit

Relevance of the criteria specified to the audit scope

Audit Checklist Development The purpose of a checklist is to provide a planned, organized method for conducting the audit and to document the results. Checklists are written by the team leader prior to conducting the audit. During the audit, the checklist allows the auditors to stay focused on the issues. However, the auditors are not limited to specific questions or items listed on the checklist but can expand the checklist, if necessary, to assure conformance to the safety issue.


The checklist is a tool meant to provide:

A planned approach to the audit

Identified areas for data collection

Focus on potential hazardous issues

A permanent record of the audit

Important information for tracking corrective actions resulting from a previous audit

Experience and judgment contributes significantly toward developing the checklist issues. In developing the checklist, the team leader must evaluate:

Constraints and exceptions for written procedures, instructions or requirements governing the issue

Regulatory requirements

Activities defined by the scope of the planned audit

All requirements to assure they are specific in nature


Chapter 4 – Internal Audit Preparation

Notification

A written notification is provided to the department manager and audit team usually including the following information:

Scope of audit

Scheduled audit dates

Pre-audit conference date

Post-audit conference date

Audit plan

Proposed Audit Checklists

Audit Outline Once the basic audit plan has been established with the affected department and notifications have been made, team leader then develops an audit outline for the audit team to follow. The audit outline is a guideline that covers the methodology the audit team will implement. It is presented at the audit team briefing and incorporates as a minimum, the following major issues:

Audit Team Briefing

Establish the audit scope

Type of audits to be conducted

Departmental requirements for performing the audit

Scheduling of audit activities

Designate audit team assignments

Preparation and list of resources

Review audit checklists

Pre-audit conference

Examination and evaluation of objective evidence

Conduct of interviews, observations, inspections, tests

Documentation of audit results

Validation of audit findings

Post-audit conference.


Audit Checklist The checklist describes specific requirements within the System Safety Program Plan, reference documentation, and regulatory requirements for the internal safety audit and usually includes:

System elements to be incorporated within the audit

Reference criteria

Sample size (as applicable).

Facility/Location to be audited

Contact person

Method of verification

Name and audit ID

Auditors.

Special Checklists In addition to the standard audit checklists, special checklists may be developed from manuals or program requirements. Such checklists provide the specific information to assist in evaluating relevant issues, which would, under normal circumstances, not be obvious. Special checklists are often used to identify the status of items on the Safety Critical Items List.

Reference Criteria As part of the audit preparation, the team leader reviews the appropriate Agency and department documents for reference criteria suitable for use in the audit plan and the checklist. The documentation to be examined during the conduct of the internal safety audit is usually listed in the System Safety Program Plan, SOP Manuals, and/or other relevant regulations, standards, codes, procedures and policies related to the scope of the planned audit, but is not limited to these areas as many functions affect or support safety. From all the reference criteria examined, certain portions will be selected for auditing and become the audit criteria. In addition, it may be appropriate to review other types of information, such as:

Previous audits involving the department

Special testing and requirements applicable to the department

New equipment/system designs

Applicable reports and analysis.


Types of Verification Documents During the conduct of an audit, sufficient information and documentation must be gathered to support a reasonable determination of the safety issue being examined. There are many different types of verification documents that may be developed, obtained or reviewed during the conduct of an internal safety audit. The team leader should request from the department manager, as early as possible, a description of what documentation and procedures will be required to be reviewed and examined during the audit. This will allow the department manager time to adequately prepare for this portion of the audit. Additional documentation may be requested during the audit by the audit team. The documentation to be evaluated during the conduct of the internal safety audit usually falls into the following types: Departmental Level Documentation Examples

Maintenance procedures

Training manuals

Proceedings of meetings

Equipment specifications

Rules /regulations

Management program plans

Chemical Inventory records Administrative Level Documentation Examples:

System Safety Program Plan

Standard Operating Procedures

Emergency Procedures

Configuration Management Plan

Hazardous Materials Management Plan

Administrative Procedures

Rule Book

Safety Rules

Fire Codes

Engineering Design Criteria

Drug and Alcohol Abuse Program

Hazard Identification Procedure

Accident/Incident Procedure

"As Built" System Drawings

Contract Specifications.


Inter-Agency Level Documentation Examples:

Fire and Rescue Agency Response Agreements

Construction Safety Manual

Contractor Safety Plan. Historical Type Documentation Examples:

Hazard Analysis

Accident Investigations

Audit Reports

Surveys

Test Results

Log Books

Maintenance Inspection/Repair History

Data and Information Evaluation The selection of documentation during the audit is only one aspect of the important process to gather evidence on safety issues. In addition to written documentation that can be reviewed during the audit, other data and information can be acquired through inspections, observations, interviews and tests. This information is transcribed onto appropriate forms or field notes to be used to support the audit checklist. The importance of data and information to the audit usually increases when the audit involves constraints, such as:

Unavailability of documents for review

Lack of management controls established

Reliability of the information

Availability of the information

Limited scope of the documentation and information available for review.


Chapter 5 – Internal Audit Execution

Audit Process Overview The Internal Safety Audit is performed in accordance with the System Safety Program Plan, procedures, Audit Plan and checklist related to the safety elements to be audited. It is a process of examination of objective evidence to determine compliance with the system safety program plan, reference documents, and federal, state and local criteria. The major issues and activities involved in performing the internal safety audit include:

Examination of documentation

Observation of equipment, facilities, and in-process tasks

Inspection of system operation, and employee workplace

Interviews with management, supervision, employees or contractors

Witnessing tests of safety critical items During the audit, the audit team should have authority to:

Access records and facilities

Issue reports

Analyze and evaluate safety data and information collected

Recommend corrective action plans.

Audit Techniques Auditing techniques require knowledge and expertise in:

Management process control methods

Interviewing methods

Performing field evaluations

Effective communications

Maintaining objectivity

Analyzing data/information

Familiarity with system operation & support functions.


Pre-Audit Conference The purpose of the pre-audit conference is to establish a positive working environment between the team leader, audit team and departmental staff and to clarify the issues for conducting the audit. It is important that all department managers and staff have a clear understanding of what safety issues will be discussed during the audit. The pre-audit conference for an internal audit conference is usually an informal discussion to coordinate activities, adjust the schedule and define approaches to be used. The pre-audit conference is held before the audit activities actually begin and should be attended by all department management staff who will be directly involved in the audit process. The team leader should prepare an agenda for the pre-audit conference that addresses all important issues of the audit plan, including:

A discussion of the areas to be audited

The purpose and scope of the audit

Previous audit findings and corrective action

Review types of documents to be examined and facilities to be inspected

Identify department coordinator, as applicable

Authority for conducting audit

Audit team experience

Background for the internal safety audit

Reference to the agency's system safety program plan and checklist

Reference document used in developing audit checklist

Review of checklist

Discussion of audit process involving:

post-audit conference

interviews process with management

examination of documentation and records

"in-process" evaluations.

method for retention of documentation

Review of audit schedule

Review of logistics for multiple locations

Document Review The department manager should be advised as to the documentation required for review by the audit team needed to establish verification for the audit element. The documentation should be evaluated for compliance with criteria described in the Checklist to establish verification of the audit element. Documentation used as objective evidence can include a statement of fact, activities in progress, records, and reports that relate to the safety of activities.


Data Taking The audit checklist is supported by audit forms and field notes used to document and record observations, interviews, document review, tests and inspections of significant issues. The checklist provides documented evidence of issues to be addressed. The checklist is considered a guide, and any omission during the audit should be identified and the reason why it was not addressed recorded. Audit forms are used to record specific issues observed such as: the serial number of a rulebook and the locations of the observation, for example, which is then summarized on the checklist.

Audit Team Review The team leader should ensure that sufficient time is planned into the schedule for reviews before the post-audit conference. These types of reviews involve discussion of observations and findings to assure they are substantial and satisfy the value of quality and the value of quantity. Although each member of the audit team should evaluate their own observations there should be frequent exchange of information between the audit team members during the audit. This allows adjustments to the schedule to re-examine an area of the audit that might be in question.

Post-Audit Conference The purpose of the post-audit conference is to ensure that the audited department is presented with all results, both positive and negative. The Team leader is responsible to assure the attendees understand all the findings, recommendations, and observations. The post-audit conference is usually performed at the conclusion of the audit and attended by the audit team and the audited department. During the post-audit conference, the team leader should:

Express appreciation for the help and support provided by the audited department

Provide a summary assessment of the internal safety audit that includes positive as well as negative issues

Discuss completed checklists

Discuss observations noted during audit

Review & discuss findings and recommendations, and the basis for them

Be prepared to verify the actual nature of any discrepancies found

Identify all issues that warrant immediate attention

Keep the discussions focused on the audit results and avoid lengthy


conversations into details of the issues

Discuss plans to follow-up regarding specific finding

Discuss recommendations for possible corrective action

Discuss the process for establishing corrective action plans and investigations

Provide an objective audit summary of facts - not opinions

Review schedule for submittal of audit report

Resolve all questions, if possible.

Identify all contested items and add to the corrective action investigation

Contested Items

Although a lot of effort goes into the planning and execution of an audit, it is not uncommon to develop contested items that arise during an audit. Disputes can occur due to a lack of familiarity of the audit team members with the function being audited, a difference in interpretation of audit criteria, or simply from difference in opinion involved in the results of analysis. All those involved in the audit should be mindful that the short period of time that is used during the audit is a “snapshot” designed to achieve a high confidence rating of the system safety process, but is not indicative of how the process functions 100% of the time. When an audit item becomes contested either during the execution of the audit or as a result of the audit results, it will be formally logged and handled in a similar manner as a corrective action item. The audit does not stop once a contested item is surfaced, however the team leader is notified of the issue to allow timely resolution as the audit process continues.


Chapter 6 – Internal Audit Result Analysis

Audit Findings The basis for audit findings and recommendations are that they are accurate and support the audit team's decision that a condition adverse to the audit criteria exists. Audit findings result in corrective actions taken by the department to resolve the root cause of the potential hazard. To arrive at effective corrective actions, the facts acquired by the audit team must be accurate and clearly support the audit findings. Upon examination of the finding, the team leader may, as needed, request the responsible department to perform a corrective action investigation to thoroughly identify the problem and aid in development of a corrective action recommendation that will preclude recurrence. The team leader prepares corrective action recommendations based upon the audit findings found in the preliminary report and reviews these with the department manager prior to the Post-Audit Conference. The audit findings and recommended corrective action should be reviewed with the department manager to:

Verify the intent of the finding

Identify any potential hazard associated with the finding

Determine the finding's root cause and influencing factors

Discuss recommendations to preclude reoccurrence.

Qualifying Evidence The evaluation of the facts is the basis for determining if the issue is in compliance with the reference documentation and system safety program plan. If the audit team finds that an issue is not in compliance, the team leader reviews the audit team’s supporting evidence to affirm that decision. If the finding is supported, the team leader prepares recommendations to serve as a possible plan of action for the audited department to investigate and develop a corrective action plan. Based upon the audit team's preliminary report, the team leader considers a variety of facts determined during the internal safety audit to establish the basis for the findings. This process includes a review of the checklist information leading to the finding, including:

Observations of in-process functions

Document reviews


Examination of data and record files

Evaluating operational systems

Interviews and meetings

Review of external or previous audits and inspections for collaboration with audit team’s results.

Quantifying Results

The team leader may see a need to quantify the evidence that was gathered. The purpose of this action is to ensure the integrity of the information that the finding was based upon. Several methods can be employed including:

Perform hazard analysis

Evaluate statistical sampling

Review studies and reports

Examine system modifications

Verifying data and information

Identification of Non Conformance After the discrepancy or exception has been measured for its value of quality and its value of quantity, a determination can be reached to validate the audit team results. This evaluation establishes the non conformance and initiates the formal logging and identification of the non conformance item as an audit finding for corrective action, follow-up, and closeout.

Hazard Analysis of Non Conforming Items

Once a non-conforming item is identified as a finding, a hazard analysis is performed to determine if the item results in a hazard to the system and what risk it represents. The analysis is performed using the Mishap Risk Assessment Values Matrix adopted from MIL Standard 882D and detailed in APTA’s Safety Management Program Plan. The purpose of the analysis is to compare the hazard severity to the hazard frequency to determine if the associated risk is High, Serious, Medium, or Low. High and Serious rankings represent Category I and II hazards.


Proposal for Corrective Action

Audit findings result in preparation of a Proposal for Corrective Action that is transmitted to the audited department. The proposal contains the description of the finding, relevant background information on the process that led to the finding, the criteria or reference documents involved and, in most cases, a recommended action plan to correct the discrepancy or exception if it has not already been adequately addressed. The audited department may be requested to perform an investigation and propose its own corrective action plan. The team leader prepares the proposal to provide a factual account of the audit finding including:

Audit checklist number or item

Reference documentation

Requirements which have not been met

The degree of discrepancy from the audit criteria

Potential effect on safety issues being evaluated

Finding and need for root cause investigation

Recommended date to be completed

Re-audit requirement (if needed)

Investigations It is the responsibility of the audited department to resolve any discrepancy by conducting an investigation to determine root cause, extent, and the potential safety hazard associated with the findings. Upon conclusion of the investigation, a corrective action plan is developed to correct the cause and prevent recurrence.

Prioritizing Corrective Actions

It is recognized that an audit may result in multiple findings for which corrective action will be requested. It will become the duty of the audited department to submit in their corrective action plan how each of these corrective actions will be implemented with the focus on controlling the Category I & II hazards as first priority. During the review and acceptance phase of the department's corrective action plan submittal, the proper prioritization of the action items will be evaluated.


Trend Analysis

Performing a trend analysis of the audit program data provides the ability to determine the effectiveness of the organization in managing its activities and improving its proficiency in safety efforts. Completing a trend analysis on the program elements provides a summary of compliance rates, areas of violation, schedule adherence, and summary of corrective action plan activities. Trend analysis is performed to monitor the following key areas:

Compliance rate by audited area

Discrepancies by audited area

Rate of Internal Audits performed

Discrepancies by location

3 year audit cycle compliance

Open corrective action items


Chapter 7 – Internal Audit Evaluation

Review Audit Goals and Objectives Upon completion of the Preliminary Audit Report, the team leader will provide a written review of the audit goals and objectives. This review will be based upon the audit plan and procedure developed for the conduct of the audit. The review will evaluate the performance of the audit and the effectiveness of the workings of the audit process.

Correspondence with SSPP The audit is also subject to evaluation against the applicable sections of the System Safety Program Plan to assure that the SSPP effectively addresses the issues that were audited. This evaluation allows the opportunity to adjust the SSPP as needed and to make recommendations to that effect.

Conditions for Re-auditing The audit evaluation may reveal the need to return to an audited area to perform a spot or targeted audit whenever the audit team does not find enough information to support verification or validation. Re-auditing may also be considered after a corrective action plan has been completed to validate the results as being in conformance.


Chapter 8 – Internal Audit Reporting

Report Process All audit activities and findings are documented, summarized, and issued as part of the audit report. The report also provides a description of the status and effects of the audit findings, discrepancies, and exceptions made including recommendations for establishing a corrective action plan. In order for an internal safety audit to be effective, the results of the audit must be used for establishing corrective action plans. For this purpose, a Preliminary Report is issued with a proposal for corrective action plans. The audited department is assigned responsibility for coordinating the follow-up to the Preliminary Report findings by conducting a corrective action investigation and establishing corrective action plans to submit to the Team leader for review and approval. The Final Audit Report, incorporating the department corrective action plan, is signed by the General Manager/CEO and formally distributed to management and the regulatory agencies where applicable.

Preliminary Report If a preliminary report is required, the team leader is responsible for preparing the report with findings and proposal for corrective action within a specified period of time following the exit conference.

Transmittal of Preliminary Report The audit report is usually sent with a letter of transmittal that includes reference to the internal safety audit by:

Name or number

Location

Date

Summary of audit results and findings

Proposal for corrective action plan

Requested date of response


Response to Preliminary Report The department manager has a specific period of time to respond to the Preliminary Report. The department response to the audit report includes but is not limited to:

Concurrence with findings

Identification of contested audit items

Plans for actions to resolve findings, discrepancies, exceptions and contested items

Results of investigation

Verification of corrective actions completed

Final Audit Report

Overview The final audit report should be prepared within a specified period of time following the audit exit conference and should include:

Scope of audit

Location and dates of audit

Description of facility/location

Audit team

Departments audited/staff interviewed

Program requirements for each checklist issue

Audit areas certified in conformance

Audit elements certified in compliance

Effective practices observed; follow-up of previous audit findings, etc.

Findings, discrepancies and exceptions

Summary of audit results

Approved corrective action plans/reports

Audit data; types of evidence examined (i.e. dwg, specs, logs, transmittals)


Completed checklists

Activities audited

Observations, interviews, inspections, and tests conducted

Status of the audited system safety program plan elements

Team leader's signature


Audit Summary The audit summary provides a description of the results:

Areas and safety elements identified as compliant

Areas and safety elements corrected since the previous audit

Description of positive activities

Audit findings and noteworthy observations

Corrective action plan summary from all departments

Audit Results There should be agreement between the team leader and the department manager concerning the validity of the results. The team leader should always report both positive and negative results. Those audit areas that fulfill requirements will result in a Certification of Conformance. The written findings for non-conforming elements must be:

Clearly stated and related to established criteria.

Describe the discrepancy and provide substantiating information.

Corrective Actions Because the internal safety audit requires specific action plans for correcting or minimizing any deviations from compliance, it is imperative that proper corrective actions be prescribed, implemented, and tracked as part of this process. The team leader distributes the Final Report containing the corrective action plan to senior management and the general manager who may assign additional responsibilities for corrective action. The corrective action process starts with an audit finding. The team leader prepares a proposal for corrective action and submits this to the department as part of the Preliminary Report. The department manager reviews and/or investigates the audit findings to:

Determine what actions should be taken

Establish priority

Allocate resources and time for completion of actions

Assign responsibility.


Corrective Action Plans Corrective action plans are a management process to resolve potential hazards identified by the audit finding and prevent their recurrence. The corrective action plans are the most important result of the safety audit in that it provides documented commitments by management for remedial action to identified deficiencies and follow-up for closure of discrepancies. The department manager's responsibility is to:

Conduct the investigations

Develop the corrective action plan

Implement the corrective action plan. An effective corrective action plan includes the following elements:

Result of the investigation of the reported discrepancy

Identification of root cause

Steps to be taken to resolve discrepancies

Resources needed to perform each task

Personnel assigned to each task

Prioritization of tasks

Target dates for completion of each task

Processing monthly status reports for each open task

Method of verification to close out each task

Transmittal of Final Report After the Team leader has received responses to the Preliminary Report, the Final Audit Report is prepared and submitted to the General Manager/CEO for review and approval. It is then returned to the Team leader for internal circulation to the departments affected and a copy is formally submitted to the regulatory agency, if applicable.


Chapter 9 – Internal Audit Tracking

Scope The team leader is responsible for establishing an audit tracking system. The tracking system provides management with the status of the audit program, schedule, and corrective action plans. It also identifies the documentation used to verify appropriate corrective actions were completed to closeout the audit finding. The team leader tracks implementation of corrective action by:

Reviewing all follow-up activity identified in the corrective action plan

Documenting results of follow-up actions

Issuance of response to audited department acknowledging acceptability of documentation

Maintain a log of closed-out actions

File audit correspondence in safety department file

Distribute completed audit report internally [NOTE: This distribution will include State Safety Oversight Agency for a Rail Transit Agency, e.g., CPUC.

Responsibilities Open discrepancies in audit findings are tracked by the team leader. Follow-up action is taken to verify that corrective action is accomplished as scheduled. Results of the follow-up are monitored and updated by the team leader. Management reports are issued monthly identifying the status of open items and any failure to comply with corrective action schedules. The department manager follows-up the findings in the audit report by:

Submitting a written monthly status report on the corrective action plan

Reporting activities that have been accomplished on tasks identified

Assure closure to the findings

Review revised documents

Verification that corrective action has been completed


Status Reports Status Reports are submitted each month to maintain tracking of activities, cost, and schedule adherence in correcting hazards identified. The team leader prepares the status reports from information available from the departments assigned to make changes. Transmittal forms are generally used for this purpose. The updated information is then entered in a database and an executive summary is prepared to advise the General Manager/CEO and the regulatory agency (as applicable) the status of progress in closing out audit deficiencies. The Department Manager may request adjustments to the adopted corrective action plans. Should the General Manager/CEO concur with the request, the Team leader will resubmit the corrective action plan, modify the Final Report, and distribute the changes to the agency and the regulatory agency (as applicable).

Tracking System The team leader maintains a formal follow-up tracking system that identifies status of all internal safety audit findings to assure corrective actions are completed. The tracking program describes the action taken to resolve and correct the finding. The information contained in the tracking system includes:

Description of the corrective actions.

Target date for completion of all actions.

Responsibility for completion of actions.

Method for periodic reviews of implementation for corrective action.

Verify that a corrective action has been accomplished as scheduled

Determine that the action was effective in preventing recurrence.

Schedule Adherence Schedule adherence for the corrective action plans are tracked on a monthly basis to ensure critical completion times are achieved. A monthly project schedule report is circulated to all involved to identify activities performed, percent complete, and time remaining to achieve goals. When needed, the Department Manager will request adjustments to the corrective action plan schedule. Should the General Manager/CEO approve the request, the Team leader will modify the Final Report and reissue the changes to the agency and the regulatory agency (as applicable).


Audit Records The internal safety audit procedures include requirements for collection, storage and maintenance of internal safety audit records. The types of audit records subject to this requirement include the documentation and information to substantiate the audit findings and verify that the internal safety audit was performed as prescribed in the System Safety Program Plan. The audit records that provide documented evidence of the internal safety audit program include:

Audit schedules

Notifications

Audit plans

Reports

Checklists

Audit notes

Correspondence


Objective evidence

Supplemental forms containing the findings

Corrective action plans and resolutions

Follow-up and verification of corrective action. The team leader is responsible for the collection, storage and maintenance of the audit records. Audit records will be kept for a defined period of time before they can be disposed. This is an agency imposed requirement and usually represents the time to complete two audit cycles to provide historical data for review. Local requirements may vary.