Practical steps to proactively measure safety performance

Canada ChapterSystem Safety Society

April 19, 2012Presented by

Terry Kelly & Tom Moir

SMS Aviation Safety Inc.

Outline

Introduction and assumptions

Steps in developing a measurement

protocol

A case study

Concluding thoughts

Assumptions

We are dealing with complexly inter-related systems People, equipment, organization (s)

Continuous & dynamic state(s) of change Goal is to explicitly manage risks to ALARP

(performance-based) ALARP is achieved through a blend of

prescribed and performance-based requirements/standards

Achieving ALARP

Five forms of functionality:1.Proactive & comprehensive2.Reliability3.Explicit safety-risk management4.Safety culture5.Safety measurement

Requirements (1)

Measures (system) performance, not just processes

Measures positive performance, as well deficiencies in performance

Is common across programs and functions in a company/organization

Focuses mainly on proactive indicators of performance, as well as reactive measures

Focuses on hazards as well as risks

Requirements (2)

Accounts for human and organizational factors as well as technical and operational factors

Evaluates the results of day-to-day decision-making, as well as strategic decisions

Discriminates in terms of safety significance Information is easily/efficiently collected and

applied The results can be replicated, and have

validity over time

Building a Measurement Tool

FOUR EASY STEPS!1.Define performance requirements2.Identify individual & organizational behaviours/outcomes3.Choose a measurement scale4.Determine collection methods

1. Define Performance Requirements

Research: Industry standards & regulations Industry association literature (e.g.,

good practices) Texts on risk management in complex

systems (e.g., Reason) Management and leadership texts

Use knowledge and experience

Safety Culture

Everyone in the company understands their role and is committed to sharing information so that the risks related to organizational, human and operational factors are actively managed. Proactive safety management is embraced throughout the organization. There is a free flow of safety-related information vertically and laterally, within and outside the organization. The positive safety culture provides organisational resiliency, which enables the organisation’s SMS to remain appropriate and effective during times of change.

2. Identify Behaviours and Outcomes

What are you going to look for? How people behave

e.g., pre-job briefings, raise hazard reports, follow procedures

How people thinke.g., company priorities, risk

How the companies manage operationse.g., metrics, documents up to date and

accurate

Function 4 - Safety Culture 

A. Safety information is valued and used at all levels of the organisation. It is obtained from consistent, unfettered and active (3-way) communication throughout the organisation.

B. Safety information is actively sought and communicated with stakeholders, including the regulator.

C. Staff have confidence in non-punitive reporting.

D. Managers trust and employ validated, safety-related information from all sources

E. Everyone from all levels “walks the talk” and are actively engaged in proactive safety management.

Caution

You don’t necessarily want to spell out too precisely the actions or outcomes you will see.

Why not?

3. Choose a Measurement Scale

Rating scale Allows you to assess change in

performance over time Allows evaluator to translate and

solidify an otherwise ambiguous perception

Rating scale needs to be accompanied by guidance

Evaluation Table

FunctionNot

Present

Exists in Part

Exists Completely

Observations / Evidence (cite sources)

 Proactive & comprehensive safety management  

1 2 3 4 5 6 7 

a. Performance-based safety policy (e.g. strategic safety goal) and principles (e.g. sharing and using safety-related information) guide the organisation’s direction, and its staff’s behaviours

☐ ☐ ☐ ☐ ☐ ☐ ☐

 

b. Safety goals & objectives are explicitly set (safety planning) and measured, and progress reported (accountability), to continuously improve safety management and performance.

☐ ☐ ☐ ☐ ☐ ☐ ☐

 

Function 1 - Proactive & Comprehensive Safety-risk Management.

All components of the organisation are actively engaged in or support the proactive management of safety-risks (e.g. operational, technical, financial, HR components, etc.). Safety-risk management is top-down, and integrated with strategic, business and HR planning….

Caution

Watch for prescriptive tools masquerading as performance-based tools!

What’s Wrong with This Picture?

4. Determine Collection Method

Choose from a combination of: Document & Data review Interviews (management & staff) Observation (of work & meetings)

Choose sample sizes to ensure validity

Multiple lines of evidence

Report

Performance ratings supported by evidence

High-level strengths System Safety Deficiencies

(prioritised in terms of risk)

Case Study

Offshore helicopter operator SMS (not regulated) in place since

2009 3-day SMS Evaluation

Function 4 C

c. Staff have confidence in non-punitive reporting

Score

             

1 2 3 4 5 6 7

Supporting evidence:•There were indications that staff are comfortable raising items in the event reporting program when they do not feel they are getting results through other avenues.•Several interviewees expressed that they did not feel their careers would be jeopardized in any way by their raising safety related concerns.•Several persons interviewed expressed the opinion that AMEs were less trusting of the non-punitive reporting program than other people, and that they were also less likely to raise event reports. This is in contrast to operations, where reporting was felt to be strong. •Management expressed the opinion that people’s attitudes were shifting (for the better) since the introduction of the SMS.

Function 5 C

c. System safety deficiencies are assessed in terms of safety significance to guide priorities to improve safety management/performance

Score             

1 2 3 4 5 6 7

Supporting evidence:•Company practices do not rigorously identify and address organizational factors. •This weakness has been recognized by company management.•See also 2.3.2.b. and 2.3.2.f.

System Safety Deficiencies

Robust Analysis(B – 2)Presently, all events reported through the electronic system receive some form of analysis. This normally consists of company management discussing the event, either in person or over the Internet, and developing action plans. More significant events receive more in-depth analysis based on Tap-root terminology or the S-H-E-L(L) model. However, these analyses do not thoroughly assess HF or OF. This consistently results in safety actions that target the behaviours of frontline workers, often by calling for increased vigilance.

Key Points

Measures (system) performance, focussing explicitly on an organization’s capability to manage risk

Focuses mainly on proactive measures, using multiple lines of evidence drawn from different types of data

Conclusion

Our objective has been to demonstrate how a simple tool can be developed and employed with

relative ease to proactively measure the safety performance of even a large and complex organization