Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved. Effective Incident Investigation SACHE Workshop September, 2005

Effective Incident Investigation

SACHE Workshop September, 2005


• On February 1, 2003 the Space Shuttle Columbia was returning from space.

• Listening to the radio in my car and heard that NASA Houston had lost contact with the Columbia.

• Needless to say I got a very queasy feeling in my stomach.

• Being a technical person I knew what that meant.

• Still kept hoping for a miracle.

• Finally NASA Houston declared a Contingency


• What is a Contingency?

• Start of an investigation

• Impact on the Flight Controllers in Houston

» Ensure their logs are up to date

» Keep hands off all switches, knobs and controller

» Computer data was impounded

• The same thing occurred at the Kennedy Space Center


• Later declared an Expanded Contingency

» All data at the suppliers to NASA impounded.

• Shuttle parts recovery

Problem was public picking up the pieces

NASA quickly made the point that this was illegal and mobilized the personnel needed for the task.

Got public participation in locating the debris but leaving it in place.


• Process Plant Incident Investigation.

• Why?

» Incident Investigation to determine the Root Cause

» Litigation Support

» Lack of Precise Information


• Focus will be on a large incident

• However scale this back to meet the event you have to deal with

• Goal is to always have the information to conclusively determine the route cause.

• If there were fatalities; litigation will follow.


• The Scene

» Plant damaged due to explosion and fire

» Control Room damaged

» Fragments are inside and outside the facility boundaries.

» Unfortunately have a fatality and multiple injuries.

» Emergency response actions have been completed


• The plant has to take the lead until the Incident Investigation Team can arrive on the site and get organized.

• ACTIONS REQUIRED

» Rope off the area of the incident including the debris field.

» Get short statement from the operating and Emergency Response personnel involved.


» Protect records from damage by the weather.

» Prepare an area for the corporate investigation team to work.

» Nominate one person from plant management and one operator minimum to participate in the investigation.

» Prepare areas for OSHA and the CSB to work but away from the investigation team

» Arrange to provide the investigation team legal counsel.

» The balance of the team has to arrive on site in < 24 hrs.

ACTIONS REQUIRED


• Investigating Team Arrives on Site

• Plant Responsibilities

• Immediate Investigation Team Goals

» Organize

» Take Control

» Start work


• Plant Responsibilities to Investigation Team Members

• Plant Orientation – processes, hazards, etc.

• Safety Equipment

• ID Cards and Plant Access

• Must be done quickly

• Accounting


• Team Requirements at Incident Site• Large meeting room with tables and chairs

• Computers/printers/copy machine(s)

• Access to plant network

• Rooms for interviews

• Recording Blackboards

• Refreshments – coffee, sodas, snacks etc.

• Lunch Service

• Unrestricted telephone access on several lines.

• Stationery supplies


• Process Information

• Mechanical

• Sampling

• Interviews

• Special Assignments

• Secretarial Support

Team Organization


• Who should be on the team?

• Not someone who would have a conflict of interest

• But:» Process Engineers

» Mechanical/Maintenance Engineers

» Operations Personnel – Supervisors, Foreman, Operators

» Chemists – QC, Analytical, R & D

» Control/Instrumentation Engineers

» Outside Consultants

» Legal Counsel

Team Organization


• Process Information

» Operating data from the Control Room

» QC data from Control Room and Lab

» Maintenance information from the Control Room

» Engineering records

» Process shutdown check

» Construct the incident time line and analyze

Team Organization


• Mechanical

» Failed Equipment

Recover the pieces and then reconstruct

» Process equipment

Protocol on how equipment/instruments will be disassembled

Storage location

Equipment/instruments to be disassembled and inspected or tested.

Coordination with the Sampling & Process Team

Team Organization


Sampling

Sampling Protocol including chain of custody

Hazards/PPE required

Storage for samples.

Sample material left in process lines or equipment

Coordinate with mechanical

Team Organization


• Sample Size

Take one large sample then divide to provide all the needed samples.

• Sample analysis

On site, Research lab, Contract lab?

• Sample shipping

Hazardous materials, need expertise

Team Organization, Sampling cont.


• Interviewing» Identify people to be interviewed

» Read the initial statements

» Develop a list of questions

» Develop an interview schedule.

» Will take longer than expected.

Team Organization


• Special Assignments» Coordinating with the Regulators» Keeping upper management informed» Keeping other stake holders informed

Others operating the process Customers

» Coordinating plant issues Release of the area Cause of the accident Meeting the teams needs

Team Organization


• Daily Team Meetings:

• Morning: Groups tell the team what their plans are for the day.

• Afternoon: Groups report to the team what they have accomplished and new information.

• Distribute/circulate completed interviews to the team.

Team Organization


• Basic Process Control System (BPCS)

» Questions:

Can the information be retrieved?

What were the control and alarm points?

What were the emergency shutdown conditions?

Actual software in the system?

How is the operating data recorded?

Alarm record?

Special Challenges


• BPCS Continued

» Answers needed about the BPCS

Could a software error or corrupted software have caused the incident?

Was the software in the system what was supposed to be there?

Special Challenges


• Interview Process

• Initial statement: KEEP IT SHORT!

» Where were you when the incident happened?

» Who were you with?

» What you saw, heard, smelled or felt?

» Time you think the incident occurred?

» What you think might have caused it?

Special Challenges


• Interview Scheduling

» Allow time for the interview

» Allow time for the write-up after an interview completed.

» Keep people to be interviewed from sitting in a waiting room with others.

Special Challenges


• Interviewing Techniques» Neutral and comfortable location

» Two interviewers – one who is familiar to the person

» During interview one asks the questions and the other records the answers.

» Sit in circle

» Explain the interview process and confidentiality

» Final Question: “Do you have anything to add or tell us?”

» Review notes to ensure no misunderstanding

Special Challenges


• Witness Questioning» Start with personal background information

Questions about the individual

Questions about family

Employment History

» Standard prepared questions

» Witness tailored questions

» Final Question: “Do you have anything to add or tell us?”

Special Challenges


• After the Interview

» Write-up the interview

» Allow the witness to review, correct, or add more

» Share the interview with the entire team

» Some contradictions are to be expected

» Arrange follow-up interview if necessary

Special Challenges


• Dealing with Regulators - Interviews

» May want to interview witnesses before you.

» Attend

» Take notes

» Use the notes as input for the team interview.

Special Challenges


• Dealing with Regulators – Evidence

» Their interest is to keep the site intact for their investigation.

» Will need an attorney present when you deal with them.

» Will have to get them to accept your protocols for evidence collection.

» What information do they want?

» What information might they have that you want?

Special Challenges


• Photography

• Using film

• Digital technology

• Photographic log book

» Roll number – memory card number

» Frame number – date and time

» Item photographed

Special Challenges


• Get the data correctly and completely

• Resist the temptation to jump to a quick conclusion or root cause.

• Stay open minded and only rule out a possible cause that the data does not support.


• Determination of the Root Cause

• Investigation team brainstorming sessions

• Develop a Fault Tree (Qualitative)

• Top event is the accident


• All possible causes that could lead to the top event are listed.

• Then work on these events, always going deeper, keep asking how or why? Identify other conditions that have to occur to make the cause feasible.

• Remember: “The only dumb question is the one not asked.”


• May identify other information needs that have been overlooked.

• Identify additional questions for interviews

• Calculation that may be needed or experiments that should be made.

• Can include management systems and human factor issues in the Fault Tree.

Why do this?


• AND

» These are two events that would have to occur simultaneously to make the above event occur.

• OR

» If either of two or more events or conditions occur the above event could occur.

Fault Tree Events


• Use of the Fault Tree

» Hang the Fault Tree on a large piece of paper in the room

» As evidence comes in mark it on the tree to indicate what is true and not true.

» This will allow determination of the Root Cause


• In summary an Incident Investigation is like a project:

• It requires organization and planning

• To get it right you have to pay attention to the details.

• Have to always question what is found or learned.

Documents

Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved. Effective Incident Investigation SACHE Workshop September, 2005