Type B Accident Investigation Board Report Chiller Line Rupture … · 2014-04-17 · Accident Investigation Board (Board) ... knowledgeable of the operations of the mechanical systems,

Final Report February 1998

Type B AccidentInvestigation Board Report

Chiller Line Ruptureat

Technical Area 35,Building 27

Los Alamos National Laboratory

Albuquerque Operations Office

This report is a product of an accident investigation board appointed by Bruce G.Twining, Manager, Albuquerque Operations Office, Department of Energy.

The Board was appointed to perform a Type B Investigation of this incident and toprepare an investigation report in accordance with DOE Order 225.1A, AccidentInvestigations.

The discussion of facts, as determined by the Board, and the views expressed in this reportdo not assume and are not intended to establish the existence of any duty at law on thepart of the U.S. Government, its employees or agents, contractors, their employees oragents, or subcontractors at any tier, or any other party.

This report neither determines nor implies liability.

i

TABLE OF CONTENTS

ACRONYMS................................................................................................................. iiiINTERPRETATION OF SIGNIFICANCE .................................................................... ivEXECUTIVE SUMMARY ....................................................................................... ES-11.0 INTRODUCTION............................................................................................... 11.1 BACKGROUND ................................................................................................. 11.2 FACILITY DESCRIPTION ................................................................................ 11.3 CHILLED WATER AND SUMP SYSTEM OPERATION ................................. 31.4 SCOPE, PURPOSE AND METHODOLOGY..................................................... 42.0 FACTS AND ANALYSIS ................................................................................... 52.1 ACCIDENT DESCRIPTION AND CHRONOLOGY, EMERGENCY

RESPONSE, INVESTIGATIVE READINESS, AND SITE CLEAN-UP............ 52.3 EXTENT OF WATER DAMAGE....................................................................... 92.4 MANAGEMENT SYSTEMS............................................................................ 102.4.1 Policy and Procedures........................................................................................ 102.4.2 Roles and Responsibilities .................................................................................. 122.4.3 Occurrence Reporting and Lessons Learned....................................................... 142.4.4 Chilled Water and Sump System Condition and Maintenance ............................. 162.4.5 Radiological Source Control .............................................................................. 172.4.6 Surveillances...................................................................................................... 182.5 ACCIDENT ANALYSIS................................................................................... 202.6 BARRIER ANALYSIS...................................................................................... 222.7 CAUSAL FACTORS......................................................................................... 253.0 CONCLUSIONS AND JUDGMENTS OF NEED............................................. 284.0 BOARD SIGNATURES.................................................................................... 30

ii

LIST OF FIGURES

Figure 1 Schematic of CWE-1

Figure 2 Photograph of CWE-1

Figure 3 Sump Pumps (PS-1 and PS-2)

Figure 4 Ruptured Coils of CWE-1

Figure 5 Thermostat Set at 32 Degrees Fahrenheit

Figure 6 Thermostat with Note to Set at 55 Degrees Fahrenheit

Figure 7 Debris Floating in Water

Figure 8 Office in Sub-Basement

Figure 9 Debris in Sub-Basement

Figure 10 Damaged Transformer

Figure 11 Radiological Source Storage

Figure 12 Actuation Arm for Sump Pumps

Figure 13 Events and Causal Factors Chart

APPENDIX A Appointment Letter

APPENDIX B Barrier Analysis

iii

ACRONYMS

AHA Activity Hazard AnalysisAL Albuquerque Operations OfficeBIO Basis for Interim OperationsCAS Condition Assessment SurveyDOE U.S. Department of EnergyEM&R Emergency Management & ResponseFM Facility ManagerFMC Facility Manager CouncilFMU Facility Management UnitFR Facility RepresentativeFSS Facilities, Safeguards & Security DivisionHC hazard categoryJCNNM Johnson Controls Northern New MexicoLAAO Los Alamos Area OfficeLANL Los Alamos National LaboratoryLIG Laboratory Implementing GuidesLIR Laboratory Implementation RequirementsLPR Laboratory Program RequirementsLS Laboratory StandardsM&O Maintenance and OperationsMEL Master Equipment ListNIS Nonproliferation and International Security DivisionORPS Occurrence Reporting and Processing SystemPPE personal protective equipmentPTLA Protection Technology Los AlamosSNM special nuclear materialSSC structures, systems and componentsTA Technical AreaUC University of California

iv

PROLOGUE

INTERPRETATION OF SIGNIFICANCE

The large property loss of $3.2 million at Los Alamos National Laboratory onNovember 17, 1997, was a result of failure by the University of California to protect theDepartment of Energy’s assets. A chiller line ruptured because of freezing temperatures,and the water collected in the sub-basement of a building because of the failure of thesump system. As a result of the flooding there was a total loss of the contents in the sub-basement and damage to the building. Inadequacies in the Laboratory’s maintenanceprogram and lessons learned program contributed heavily to this incident.

Although the facilities and equipment were considered by line management to be old anddeteriorating, adequate assessments were not made to determine the consequences tomissions should equipment fail. Even though there were institutional maintenancestandards, they were guidance and were not required to be implemented by linemanagement. As a result, a freeze protection plan was not implemented. A completeapproach to maintenance by the Laboratory is needed that emphasizes implementation ofrequirements and procedures, individual and line responsibility and accountability,effective training, and thorough oversight and feedback to management.

To prevent recurrence, line management must learn from previous incidents. Althoughinformation concerning other freeze protection incidents were known by the Laboratory,they did not ensure that the applicable lessons learned were implemented institutionally.

ES-1

EXECUTIVE SUMMARY

INTRODUCTION

A property damage incident which occurred at the Los Alamos National Laboratory(LANL), Technical Area 35 (TA-35), Building 27, was investigated since the estimateddamage was expected to exceed one million dollars. During the investigation, the DOEAccident Investigation Board (Board) used various analytical accident investigationtechniques including accident analysis, barrier analysis, and event and causal factoranalysis. The Board observed the function tests of the mechanical equipment involved inthe incident, inspected the incident site, reviewed events surrounding the incident, andconducted extensive interviews and document reviews to determine the factors thatcontributed to the incident. Relevant management systems were also evaluated inaccordance with DOE Order 225.1A, Accident Investigations.

INCIDENT DESCRIPTION

The incident was discovered at approximately 6:45 a.m., on Monday, November 17, 1997,when a Protection Technology Los Alamos (PTLA) employee responded to a securityalarm. Sometime past normal working hours (after 5:00 p.m.) on Friday, November 14,1997, a chilled water system froze, rupturing its copper coil at several locations. Theincident resulted in flooding of the sub-basement of TA-35 Building 27 with 58½ inchesof water. This caused $3.2 million in damage to the facility and to the equipment used forNonproliferation and International Security operations.

LANL has ensured the safety and health of the workers during the clean-up activities.Plan of the day meetings were held each day to cover the activities along with the potentialhazards. Controls were established and verified to reduce injury and illnesses. Clean-up isongoing because of health concerns from the water damage.

CAUSAL FACTORS

The direct cause of the incident was the improper setting of the reservoir setpointtemperature, which caused the dampers to remain open during subfreezing temperatures.

The root causes were: (1) failure by LANL to implement an effective institutional lessonslearned program, (2) failure by LANL to ensure the facility management organization wasknowledgeable of the operations of the mechanical systems, (3) failure by LANL to ensurethe roles and responsibilities of the facility management organizations were clear andunderstood, (4) failure by LANL to establish maintenance requirements, and (5) failure byLANL, to provide oversight of facility management maintenance activities.

The contributing causes were: (1) maintenance categorization of equipment was incorrect,(2) maintenance was not conducted more frequently based on established criteria, (3)legacy design features were not reevaluated after the facilities and mission changed, (4)

ES-2

failure by DOE/AL and LAAO to provide oversight of the FM maintenance activities, and(5) radiological source control was not completely developed and implemented.

CONCLUSIONS AND JUDGMENTS OF NEED

Table ES-1 presents the conclusions and judgments of need determined by the Board.Conclusions of the Board are those considered significant and are based upon facts andpertinent analytical results. Judgments of need are managerial controls and maintenancepractices believed by the Board to be necessary to prevent or mitigate the probability orseverity of a recurrence of this type of incident.

Conclusions Judgments of NeedApplication of lessons learned fromfreeze protection incidents was noteffective.

LANL needs to ensure that institutional correctiveactions are uniformly and promptly implemented.

LAAO needs to establish a process to ensure importantinstitutional corrective actions, based on lessonslearned, are implemented at LANL.

FMU-75 did not develop a freezeprotection plan and LANL did notensure that a freeze protection planwas developed.

LANL needs to establish requirements for FMUs todevelop a freeze protection plans and ensure they areimplemented at all FMUs.

FMU-75 did not take actions toprotect TA-35, Building 27 sub-basement from a flood.

LANL needs to re-evaluate equipment categorizationand “legacy design” for failure modes to preventpotential flooding.

LANL needs to ensure operability of all safetyequipment including sump pumps in the basement andsub-basement.

NIS-5 did not complete plannedradiological source control actionswhich would have facilitated floodresponse and minimized flooddamage.

NIS-5 needs to complete radiological source controland storage actions.

Institutional maintenancemanagement program lacks clearrequirements, training, and oversight.

LANL needs to evaluate current requirements orestablish requirements for the categorization,maintenance, and surveillance of systems. Clear rolesand responsibilities and associated training to provideunderstanding of both responsibilities and operationsneed to be developed. Evaluation of performancethrough oversight to these requirements needs to beperformed to provide institutional consistency andincorporation of shared corporate knowledge betweenFMUs. All FMs need to be held accountable for theoperation of the facility.

ES-3

The reduction in staff and qualifiedindividuals resulted in inadequateoversight of the institutionalmanagement maintenance program.

DOE/AL and LAAO need to strategically alignresources in order to provide oversight of themaintenance management program

1

Type B Accident Investigation Board Reporton the November 17, 1997 Chiller Line Rupture

at Technical Area 35, Building 27,Los Alamos National Laboratory

1.0 INTRODUCTION

1.1 BACKGROUND

On November 17, 1997, at approximately 6:45 a.m., aProtection Technology Los Alamos (PTLA) Security Guardinvestigated a security alarm at Los Alamos NationalLaboratory (LANL),Technical Area 35 (TA-35), Building27. The alarm was initiated because of the hydrostaticpressure on the door or water damage to the electronics ofthe security alarm. Upon entry into the building, the guarddiscovered that the sub-basement level of the facility wasflooded. An estimate of the property damage associatedwith the flooding condition was initially determined to be$300,000. After additional evaluation by the user group, thisestimate increased to approximately $1,000,000 in damage.On November 21, 1997, Bruce Twining, Manager,Albuquerque Operations Office (AL), US Department ofEnergy (DOE) established a Type B Accident InvestigationBoard (Board) to investigate this incident in accordancewith DOE Order 225.1A, Accident Investigations. Theappointment letter can be found in Appendix A.

1.2 FACILITY DESCRIPTION

LANL, located in north-central New Mexico, is operated bythe University of California (UC) under contract to DOE.Its primary mission is to enhance national security byapplying its scientific and engineering capabilities to nuclearweapons technology. LANL also performs other work notrelated to nuclear weapons technology. LANL occupies 43square miles and consists of 47 active technical areasmanaged by twenty Facility Management Units (FMU).

The scene of this incident, TA-35, Building 27, is theresponsibility of FMU-75, which reports to theNonproliferation and International Security (NIS) Division.Building 27 has three floor levels where work is conducted.The main tenant is the Safeguards, Science and TechnologyGroup, NIS-5. The first floor (ground level) consists mainly

On November 17, 1997, aPTLA Security Guardinvestigated an alarm at TA-35, Building 27 anddiscovered the 3rd floor sub-basement flooded.

On November 21, 1997 aType B Accident InvestigationBoard was established by theAlbuquerque OperationsOffice.

2

of office and administrative work areas. The NIS-5 groupoffice resides there, as well as the training and publicationssection. There is one small radiological source storagerepository on this floor. There is also an equipment stagingarea termed the “high bay.”

The second floor (basement) is also primarily office space.There is one room used for archival storage and anotherused for storage of surplus electronic equipment. The mainexperimental area is the “hot cell” or shield cell. The hotcell is also used for radiological source storage.

The third floor (sub-basement), where the floodingoccurred, is used primarily for four programmatic efforts.The programs currently in progress are (1) the InertialElectrostatically Confined Plasma Neutron Generator, (2)training classes usually in gamma-ray techniques, (3)calibration activities supporting material control andaccountability programs, and (4) Remote MonitoringProject. In addition, the sub-basement is used for sealedradiological source and small amounts of special nuclearmaterial (SNM) storage. The SNM is stored inside thevault, which is a locked room inside the pipe room. Thepipe room is also locked and secure. The pipe room itself isused also for training in “hold up” measurements.Radiological sources are stored in the “cage”, which isadjacent to the sub-basement hot cell. The sub-basement hotcell is similar in design and operation to the basement hotcell.

A Basis for Interim Operations (BIO) for Building 27,which documents approval to conduct operations, wassigned by the Facility Manager, NIS-5 Group Leader andNIS Division Leader on June 19, 1996. The BIO indicatesthat Building 27 was classified as a Hazard Category 3(HC-3) nuclear facility based on a direct comparison withthe radionuclide threshold limits per DOE-STD-1027,“Hazard Categorization and Accident Analysis Techniquesfor Compliance with DOE Order 5480.23, Nuclear SafetyAnalysis Reports.” The BIO also indicates a determinationthat Building 27 had negligible onsite and off-siteconsequences for the worst unmitigated accident. Therewas no accident scenarios described in the BIO thataddressed flooding in the basement.

3

1.3 CHILLED WATER AND SUMP SYSTEMOPERATION

The chilled water system for TA-35, Building 27 provideschilled water for ventilation cooling for the first-floorcomputer room and other cooling loads within the facility.None of these loads is considered mission-critical. Aschematic diagram of the chilled water system is found inFigure 1, and Figure 2 shows a photograph of the chilledwater system. The chilled water system is a closed-loopsystem that consists of two full-capacity chilled waterpumps (PCW-11 and PCW-12), which take a suction on astorage tank (TCW-1), and pump the water through achilled water evaporator (CWE-1) and out to the chilledwater header. The return flow enters tank TCW-1 through areturn header. The tank has an automatic makeup system tomaintain a prescribed level. The pumps operate one at atime and can be manually shifted from a single controlswitch. The chilled water is cooled in the CWE-1 which islocated on the first floor. CWE-1 consists of a fan whichdraws outside air over the cooling coils. A set ofmechanically interlocked dampers are modulated through asingle temperature control sensor to maintain the chilledwater at a temperature within a 3-degree preset operatingband. The temperature is sensed from a reservoir of waterthat is below the cooling coils. A separate system sprayswater from the reservoir across the cooling coils.

The system also contains an automatic emergency “feed andbleed” mode of operation. This feature was a legacy designfrom the late 1960’s in order to provide emergency coolingto a flux reactor that was never installed. An electro-pnuematic relay, which is energized through a pressureswitch located on the discharge side of the chilled waterpumps, provides air pressure to keep shut a pneumatically-operated valve (V-1). In a normal operating condition, thisisolates the chilled water system from the industrial watersystem. If the running chilled water pump loses outletpressure, the pressure switch opens and vents the air off thevalve, allowing industrial water at a pressure of 100 psi toflow into the chilled water system through the header totank TCW-1. The water fills the tank and then exits throughan overflow line that dumps on the pavement outside of thebuilding. In this mode of operation the cooling coils have noflow due to check valves located at the discharge of the

The chilled water systemprovides chilled water for thefacility.

The chilled water systemcontains a legacy, automaticemergency “feed and bleed”mode of operation.

PC

W-1

1

PS

over

flow

mak

eup

wat

er

CC

W R

etur

n

off

- 22

#

PS

on -

30#

PSW

-3

F

CC

W H

eade

r

Indu

stri

alW

ater

E/P

Fai

l Ope

n

2 1

/2”

2 1

/2”

4”

MP

Acc

ess

Por

t

B

3/4

”

T

Inta

ke

Pos

.C

hem

Add

Exh

aust

A

~~

)

3/4

”

~

( (

PC

W-1

2T

CW

-1

CW

E-1

V-1

mak

eup

wat

er

~

~

PCW

P

ump

chill

ed w

ater

PS

Pres

sure

sw

itch

TC

W

Tan

k fo

r ch

illed

wat

erC

CW

C

old

chill

ed w

ater

PSW

Pum

p sp

ray

wat

erC

WE

C

hille

d w

ater

eva

pora

tor

E/P

Ele

ctro

-pne

mat

icPo

s

P

ositi

oner

T

The

rmos

tat

Fig

ure-

1

Chi

lled

Wat

er E

vapo

rato

r

Figure 3: Sump Pumps (PS-1 & PS-2)

Figure 2: Chilled Water Evaporator (CWE) - 1

4

pumps. When the pump discharge pressure is restoredabove the pressure switch reset point, valve V-1 shuts.

A sump system located on the sub-basement has two 80-gallon/minute sump pumps, which alternate in a first stageconfiguration and then simultaneously if water reaches asecond stage point as determined by a float in the sump pit.A shroud protects the float-activated switches (Figure 3).

1.4 SCOPE, PURPOSE, AND METHODOLOGY

The Board began its investigation on November 24, 1997,and completed its investigation on December 19, 1997. OnJanuary 6, 1998, the Board submitted its findings to the ALManager.

The scope of the Board’s accident investigation included allactivities required to review and analyze the circumstancessurrounding the accident and to determine the causes.During the investigation, the Board inspected the accidentscene and associated property damage, reviewedinformation and photographs provided by the FacilityManager (FM), observed the testing of the mechanicalsystems after power was restored, reviewed the eventsleading to the incident, reviewed the emergency responseand incident clean-up, conducted extensive interviews anddocument reviews, and performed various accidentinvestigation techniques. The Board also evaluated theadequacy of the contractor’s maintenance system.

The purpose of this investigation was to identify causalfactors of this incident, including deficiencies, if any, in theoperation and maintenance of the affected systems. Theinvestigation report will inform the DOE community oflessons learned to promote improvement and to reduce thepotential for similar incidents resulting in property loss.

The Board conducted its investigation, with focus onmanagement systems, using the following methodology:

• The Board gathered facts relevant to the incident;• The Board interviewed personnel associated with the

incident;• The Board evaluated relevant management systems and

factors, and

The scope of the AccidentInvestigation included allactivities required to identifythe cause of the accident.

The purpose of theinvestigation was to identifycausal factors in order toinform the DOE communityof lessons learned.

5

• The Board analyzed barrier and event and causal factorsto determine the causes of the incident and the causes ofthe extensive property damage.

2.0 FACTS AND ANALYSIS

2.1 ACCIDENT DESCRIPTION ANDCHRONOLOGY, EMERGENCY RESPONSE,INVESTIGATIVE READINESS, AND SITECLEAN UP

The flooding of TA-35, Building 27 was due to the ruptureof the CWE-1 copper cooling coils caused by freezing. Thecoils ruptured in a number of locations (See Figure 4).These were primarily at bends in the coils; however, anumber of ruptures were found in the center portion of thecoils. The freezing and subsequent numerous ruptures tookplace after 8:00 p.m. on Friday, November 15, 1997.Following the ruptures of the cooling coils, water flowed(estimated at 200 - 250 gallons/minute) into the mechanicalroom and primarily down the east stairwell flooding the sub-basement of the facility.

Inspection of CWE-1 the day after the incident wasdiscovered revealed that one vane of the outlet damper (seelocation A on Figure 1) was stuck in the open positionbecause of a broken linkage arm. It was also noted that thetemperature sensor for this system was set at 32 degreesFahrenheit rather than the 55-degree setting indicated on theoperator aid posted nearby (See Figures 5 and 6). Inaddition, an access port was opened above the coil stackwith the cover plate of the port lying on the ground next tothe unit. The Board determined that only the temperaturesensor setting contributed to the incident.

At approximately 6:45 a.m., November 17, 1997, PTLAresponded to a security alarm in TA-35 Building 27. Thesecurity officer discovered the flooded condition of Building27 and reported the condition to the Facility Support,Johnson Controls Northern New Mexico (JCNNM)Utilities. They in turn contacted Facility Management atTA-35 and Emergency Management and Response(EM&R), who contacted Nuclear Materials Control &Accountability Group, FSS-12; Nuclear Criticality SafetyGroup, ESH-6; Water Quality & Hydrology Group, ESH-18; and the Los Alamos Area Office (LAAO). Facility

The flooding in Building 27was discovered because of asecurity alarm.

6

Support personnel shut the water supply to the CWE-1. Atapproximately 7:30 a.m., the FM arrived at the building. Noformal incident command was established by EM&R, sincethey considered facility management to have the situation incontrol. EM&R did remain at the scene to provide anyneeded technical support.

At 7:50 a.m., electrical power to the building was turned offat the sub-station. ESH-6 began reviewing the inventory ofsealed radiological sources and evaluating potentialcriticality issues. The criticality evaluation that wasconducted revealed no concern because the plutonium wasstored in 6M drums and the uranium was mostly ofmoderate enrichment. At 8:20 a.m., a critique was heldwith specialists in electrical, mechanical, radiation,hydrology, waste treatment and waste management toidentify issues and develop plans to mitigate the incident. Atthat time, it was determined that the sub-basement heldabout 400,000 gallons of water based on a water level of58½ inches. Between 10:00 a.m. and 12:00 noon, sampleswere taken of the water in the stairwell for radioactivityanalysis. Tests of the samples indicated that there was NoDetectable Activity (NDA) for gamma, alpha and betaradiation; however, tests for tritium indicated a level of 38picocuries per liter. This level was within background levelsfor tritium (50 pCi/L). The waste treatment specialistdetermined the water could be pumped to the wastetreatment plant. Tests were also conducted to determine ifthere was any oil or other chemicals in the water so that thewater could be pumped into the sewer. Tests for theseconstituents indicated that there were no detectable levels.At the same time, dumpsters were dedicated and stationedto collect and hold the trash for release.

At 1:30 p.m., a review of the electrical concerns was heldby the FM, Division Electrical Safety Officer, SupportServices Electrical Safety Officer, and the AreaCoordinator. It was determined that a generator could beconnected to the distribution panels at the ground level toprovide electricity to the pumps. At 6:20 p.m., pumpsbrought in from off-site began pumping out the water. Allpumps were fitted with screens to ensure that noradiological sources were inadvertently discharged to thewaste treatment facility. Pumping continued until 1:40 p.m.,November 18, 1997, when the water level dropped to alevel of eighteen inches. A survey team composed of the

It was determined that thesub-basement held about400,000 gallons of waterbased on a water level of58½ inches.

Tests were conducted todetermine if there was any oilor other chemicals in thewater so that the water couldbe pumped into the sewer.

An electrical safetyassessment of the sub-basement was conducted.

Figure 4: Ruptured Coils of CWE-1

Figure 6: Thermostat with note set at 55 degrees Fahrenheit

Figure 5: Thermostat Set at 32 Degrees Fahrenheit

7

electrical safety officers entered the sub-basement to assessthe damage. All power sources were locked and tagged outbefore entry. The team also wore boots and gloves rated for1 kVa while working in this area in case there was anunknown electrical supply source which fed the building.During this assessment, the team disconnected the electricalequipment from the outlets. When surveying the closed piperoom in the sub-basement, the team noted possible evidenceof oil. Additional tests were taken for oil, grease and poly-chlorinated biphenyls, and pumping remained suspendeduntil the results of these tests were determined.

Test results were negative, so pumping resumed at 3:40p.m., November 19, 1997. Pumping continued until thewater level reached a level of one inch. On November 20,1997, personnel entered the flooded area to assess thedamage. Again, the team members wore appropriatepersonal protective equipment (PPE).

On November 21, 1997, the custodians entered the sub-basement area, wearing PPE, to remove the remainingwater. Before personnel entered the sub-basement area,power sources were locked and tagged out. In addition, thecustodial staff did not use any powered equipment whenremoving the remaining water. On November 22, 1997, theelectricians began analyzing, cleaning, and replacingelectrical distribution equipment in the flooded area. OnNovember 26, 1997, the electrical power was restored tothe building. The FM initiated clean-up and propertyinventory measures, which are ongoing, because of generalindustrial safety and health concerns.

Daily plan-of-the-day meetings were held among TA-35facility management and the clean-up workers. During thesemeetings, activities for the day were discussed, along withthe hazards and control of these hazards. The main focus ofthese meetings was worker safety and health with the goalsof pumping all the water out of the building, returning thebuilding to safe electrical operation, and verifying thelocation of all nuclear material. In addition to the meetings,the site’s work control process covered the clean-upactivities. Activity Hazard Analyses (AHAs) weredeveloped for each of the crafts involved in the clean-up.Review of these documents show that these AHAs weregeneric and did not specify the controls and PPE usedduring the clean-up. However, plan-of-the-day meetings

Plan-of-the-day meetingswere in place to ensureproper controls were in placeduring clean-up activities.

8

ensured that proper controls were in place during the clean-up activities.

There were over 200 sealed radiological sources and SNMlocated in the sub-basement of the facility. Once the level ofthe water dropped to the one-inch level, the goal was tolocate these sources. To ensure an accurate and completeaccountability of the radiological sources, NIS-5 conducteda full inventory of all radiological sources assigned to them.The radiological source removal log for the sub-basement“cage” indicated only one radiological source was loggedout at the time of the flood. It was determined that the onelogged out radiological source was a Cesium-137 sourcethat was being used in the pipe room. This radiologicalsource was not stored in the secure cage location and it wasrecognized that it would have to be searched for. It wasfound near an unscreened drain. In fact, all drains wereunscreened; however, screens were placed on the drainsafter the survey was conducted at the one-inch water level.

NIS-5 conducted an SNM inventory in the pipe room andthe vault. All SNM was accounted for immediately. Allplutonium had been stored in 6M, Type B drums and wasboth secure and dry. NIS-5 conducted a radiological sourceinventory in the cage. As a result of the flooding severalradiological sources had fallen off the shelves. Theradiological source inventory of non-SNM was incomplete,and a total of seven radiological sources were missing. Twoadditional inventory checks were performed betweenNovember 24, 1997, and December 1, 1997, before 100%radiological source accountability could be ensured.

On November 24, 1997, contamination was detected onthree personnel booties, the highest level being 1200 countsper minute (cpm); based on analysis it was believed to beCesium-137. On November 25, 1997, large area swipeswere performed in the cage and hot cell and indicated lowlevels of removable contamination. After the inventory ofthe radiological sources was completed, a leak survey wasscheduled by NIS to determine the extent of damage to theradiological sources.

Actions taken by LANL to preserve the integrity of theincident scene were effective. PTLA secured access to thesite and the FM controlled access to the building. Numerousdigital photographs were taken of the incident and property

There were over 200 sealedradiological sources andSNM located on the sub-basement of the facility.

A total of three inventorychecks were performedbetween November 24, 1997and December 1, 1997 before100% accountability could beensured.

Actions taken by LANL topreserve the integrity of theincident scene were effective.

9

damage. The evaluation of the electrical equipment loss wasnot performed until after the Board convened at the site.The LAAO backup Facility Representative (FR) respondedto the scene and returned the following day to attend thecritique. It was determined after the critique that thisincident should be classified as a Type B Accident since atthis point, the amount of damage could exceed one milliondollars. The LAAO accident investigation point of contactrequested that the contractor continue in its clean-upactivities and maintain control of workers entering thefacility. There was delay in forming the Board. Because ofthis delay, the Board did not convene at the site untilNovember 24, 1997. At that time, a briefing was given bythe FM detailing the clean-up activity. The minor deficiencyinvolving the timely selection and mobilization of the Boarddid not affect the outcome of this investigation.

2.3 EXTENT OF WATER DAMAGE

The flooding of the sub-basement of Building 27 causedminor damage to the facility and radiological material andextensive damage to experimental equipment. Figures 7through 11 illustrate examples of the damage. In the sub-basement, major portions of the electrical distributionsystem (breakers, transformers, receptacles) had to bereplaced due to water damage. Radiological sources weredamaged because of the flooding and subsequentsubmersion. Six of the sources were found to be leaking,and twenty had cross contamination. The cross-contamination was a result of the sealed source recoveryand staging for subsequent disposition. Two radiologicalsources had contamination levels of less than2200 dpm/100 cm2 for either alpha or beta-gamma. TheNIS-5 group leader indicated that all leaking radiologicalsources would be disposed of unless they could berefurbished.

A damage report compiled by FMU-75 indicated a totalcost estimate of $3.2 million for the flooding of Building 27.This report was provided to the Board on December 18,1997. The $3.2 million represents $125,000 for facility costswhich includes clean-up, emergency response, fireprotection, roads and grounds, and other miscellaneouscosts. About $1.3 million dollars is attributed to theacquisition cost of bar-coded equipment and $1.8 million isattributed to the acquisition cost of scientific equipment

The flooding of the sub-basement of Building 27caused extensive damage to thefacility, experimentalequipment and radiologicalmaterial.

A damage report compiled byFMU-75 indicated a total costestimate of $3.2 million for theflooding of Building 27.

Figure 7: Debris floating in water in sub-basement

Figure 8: An office in sub-basement, note water mark on wall

Figure 10: Damaged transformer in sub-basement

Figure 9: Debris floating in sub-basement with one inch of water

Figure 11: Radiological source storage

10

(i.e., joint research equipment with Russia). The $3.2million cost estimate assumes that the majority of allequipment located in the sub-basement was damagedbeyond repair.

2.4 MANAGEMENT SYSTEMS

2.4.1 Policy and Procedures

LANL has a hierarchy of requirements comprisingLaboratory Program Requirements (LPR), LaboratoryImplementation Requirements (LIR) or LaboratoryStandards (LS), Laboratory Implementing Guides (LIG),and in the case of maintenance activities, Maintenance andOperations (M&O) Standards. The LPR are performancerequirements that individual Division Directors are expectedto follow. The following standard statements are examplestaken from the maintenance LPR:

• Maintenance history is documented and used to supportmaintenance activities.

• Preventative maintenance is conducted in an effectivemanner and contributes to effective performance andreliability of systems, structures and components.

• Maintenance is conducted in an effective and efficientmanner.

• The condition of facility systems, structures andcomponents support safe and reliable operation.

• Each facility is managed throughout its life cycle so thatits capability meets program needs.

The LIR establishes operational requirements for Divisionsin order to provide consistency in key institutional processesat LANL. For maintenance, LS 121-01.1, Categorization ofSystems & Equipment Via the Graded Approach, and LS121-02.0, Graded Approach to the Conduct ofMaintenance, are in effect until a LIR is specifically writtento address maintenance management.

LANL Standard 121-01.1 requires that when assigningsystems to a graded approach category of consequence, theworst-case credible failure mode should be assumed. Indetermining the worst-case failure mode and credibility, thefollowing considerations are presented: failures of similarsystems, operating history, weak points in the system, andinsufficient maintenance. The categorization of each

11

identified system should be recorded in the facility MasterEquipment List (MEL). Attached to the standard is anexample listing of generic systems and equipment with arecommended graded approach category. Chilled watersystems or sump pumps are not specifically listed; however,waste systems are designated as M3. CWE-1 was classifiedby FMU-75 as a category M4 along with sump pumps PS-1and PS-2. To be classified as category M4, the failure of thestructure, system, or component would have no impact onthe public or workers, but may cause minor damage to thefacility. Category 4 equipment requires only goodbusiness/maintenance practice in order to remain operative.

LS 121-02.0 requires implementation of a maintenanceprogram based on guidance provided in the document.Facility managers are also required to grade their systems,develop equipment lists, and assign appropriate maintenanceresources. An attached matrix in the standard identifiesrequired documentation based on the grade. Thisdocumentation includes: cost identification, modificationsand additions, maintenance procedures, post maintenancetesting, surveillance, preventative maintenance, predictivemaintenance, and seasonal service and freeze protection.However, M4 equipment requires only cost identificationfor real property. Freeze protection for M4 equipment islisted as “should be considered” rather than “shall” in thisstandard.

For maintenance, LIG 207-00-01.0, MaintenanceProcedures and Training, provides guidance fordevelopment of maintenance procedures. The guide focuseson the procedures for recurring scheduled maintenance thatmay be considered preventive or predictive. The guidanceoutlines the need for formal procedures based on equipmentcategorization. If the equipment is classified as M4, thenformal procedures are not required and “Skill of the Craft”is the recommended practice. The equipment involved inthis incident is classified as M4. Although procedures arenot required, the document provides guidance indetermining the maintenance activities to be performedalong with the frequency. Consideration for type andfrequency of maintenance should include seasonal changes(freezing) and when records indicate a significant change inreliability.

12

The LANL Maintenance and Operations Standards Manualis a collection of the Maintenance & Operations Standardsthat are not specific to any single piece of equipment. TheM&O Standards statement of authority signed by theFacility Manager Council (FMC) Chairperson and theFacilities, Safeguards & Security (FSS) Division Directorstates, “This manual sets minimum maintenance andoperations (M&O) standards adopted by the FacilityManager Council … and the Facilities, Security andSafeguards (FSS) organizations responsible for M&Omanagement. In adopting these standards, the FMC andFSS Division recommends their implementation, butindividual Facility Managers are responsible for planning theimplementation of these standards at their FacilityManagement Unit (FMU).”

LANL Maintenance and Operations Standard MO 3.7-600,Rev. 0, issued on October 22, 1997, addresses freezeprotection. This standard was developed as a correctiveaction to an occurrence report (ALO-LA-LANL-1996-0007). The standard requires a freeze protection plan,itemizing property and equipment to be protected, andplanned actions with scheduled and completion dates. Thereare no examples provided in the standard to assist in thedevelopment of a plan. This standard requires tests ofsystems controlled by dampers; however, this test was notconducted on CWE-1. The FMU has only an informalfreeze protection checklist at TA-35, Building 27, but nofreeze protection plan was developed. However, there wasno inclusion of the testing of dampers and associatedcontrollers in the checklist. During the interview process,the FM stated that he was disappointed to find that therewas no official plan for freeze protection.

2.4.2 Roles and Responsibilities

FMU-75’s mission is to support the programmatic goals ofthe personnel residing in the facility by managing thephysical plant and providing Building Manager service foreach building. In order to accomplish this mission, FMU-75has created an organizational team comprising the FM,three Facility Coordinators, Area Coordinator, SafetyEngineer, and a Radiological Control Technician. The majortasks of the team include: liaison with FSS personnel formaintenance support; schedule, track, and report all facilitymaintenance activities; manage the funds for facility

The FMC and FSS Divisionrecommends implementation ofmaintenance standards .

At TA-35, Building 27, nofreeze protection plan wasdeveloped.

FMU-75 supportsprogrammatic goals ofresiding personnel bymanaging the plant.

13

maintenance; coordinate repairs and resolution ofdeficiencies that require action by support groups; overseecrafts performing maintenance; maintain work order controlsystem; plan for upgrades; serve as the point of contact forall facility modifications; initiate work orders; assure thatfacility-owned equipment is inspected as required andrecords are maintained; identify critical systems; anddevelop and implement the maintenance managementprogram for the facility. In addition, the facility managementis to conduct management walkthroughs to ensure facilitywork is conducted safely. The responsibilities of the FMwith respect to the facility infrastructure were clearlyunderstood by tenant organizations.

Per LIR 280-02-01.0, Laboratory Facility ManagementProgram, the Division Director has the ultimate authorityfor the assigned facilities. The FM is a line manager and anagent of the owning Division Director, and has the authoritycommensurate with the responsibilities delegated to him bythe owning Division Director. The FM has the sameauthority as other line managers. The FM is given thefollowing authority and responsibilities with respect tomaintenance:

• Approve any activities, or proposed changes toactivities, that could affect the established facilityoperating limits;

• Control support and services personnel who performfunctions for the facility to maintain the establishedfacility operating limits, or the facility assets;

• Determine the facility operating limits through thehazard analysis process;

• Review, approve, and assess operations within thefacility;

• Efficiently and effectively maintain the facility operatinglimits.

• Efficiently and effectively maintain the facility’sstructures, systems, and components capabilities andassets.

This LIR does not contain any requirements with respect tothe training and qualifications of the Facility Manager toperform these tasks. FSS Division is in the process ofdeveloping an LIR that would stipulate the trainingrequirements for the Facility Managers.

The Division Director has theultimate authority for theassigned facilities.

The LIR does not contain anyrequirements with respect tothe training and qualificationsof the Facility Manager toperform these tasks.

14

The Area Coordinator has the responsibility for corrective,preventative, and predictive maintenance. In fulfilling theresponsibilities of the Zone Team Leader, the AreaCoordinator provides technical leadership and advice to theFacility Coordinator on proper operations and maintenanceactivities for the facility structure, systems and componentsas well as the design and development of the MEL incollaboration with the Facility Coordinator and the FM.Periodic walkthroughs of the NIS Division facilities arerequired to monitor the effectiveness of the operations andmaintenance program.

The Facility Coordinator is a member of the FMU team.Under the direction of the FM, the Facility Coordinatorinspects the buildings and surrounding grounds for problemsand oversees the maintenance and checking of facilitysupport systems.

The LANL Facility Engineering Services Group (FSS-9)has developed a job position entitled Facility Engineerwhich is optional for the FMU to use. FMU-75 does notutilize this position. The Facility Engineer is accountable tothe FM for defining the performance, operating andmaintenance criteria, and standards for assigned facilitystructures, systems and components; providing the technicalexpertise to support the efficient and effective long-termoperation or use of the assigned structures, systems andcomponents; and providing consultation to tenants and theLANL institution regarding performance capabilities,operational capabilities and condition of their assignedstructures, systems and components.

2.4.3 Occurrence Reporting and Lessons Learned

Occurrence Report ALO-LA-LANL-RADIOCHEM-1996-0007, discovery date January 2, 1996, resulted in acorrective action to raise the setpoint that controlleddampers to close. This corrective action was developed asan additional precaution to address a freeze protectionissue. The effectiveness of the controlled dampers, whichclosed as designed for this occurrence when the temperaturereached between 40 and 45 degrees Fahrenheit, wasevaluated. The setpoint of the damper controls was raised tobetween 50 and 55 degrees Fahrenheit. The correctiveactions to this event were limited to TA-48-1 and this

The Area Coordinator has theresponsibility for corrective,preventative, and predictivemaintenance.

The Facility Coordinatoroversees the maintenance offacility support systems.

The Facility Engineer which isnot utilized by FMU-75 isaccountable for maintenancecriteria.

15

specific piece of equipment. There were no institutionalcorrective actions for this event.

During the week of December 16-20, 1996, LANLexperienced subfreezing temperatures that resulted in aseries of freeze protection events that were “rolled-up” intooccurrence report ALO-LA-LANL-1996-0007. The eventswere categorized as a reportable occurrence on December19, 1996. The notification report for these events wassubmitted to the ORPS system on December 23, 1996. Thefinal report was not submitted to ORPS until October 30,1997. This exceeds the 45 day limit set by DOE Order232.1, Occurrence Reporting and Processing of OperationsInformation, for submitting final reports. Through theDOE/UC contract, DOE has recognized that finalization ofoccurrence reports in the 45-day limit is a problem. Duringthe last performance period, LANL did not meetexpectations and issued a final report within 45 days only52% of the time.

In addition to not meeting the 45-day limit, finalization ofthe report in time to address seasonal considerations wasnot accomplished. The stated corrective actions (2 & 3)were not completed until September 30, 1997, andOctober 22, 1997, respectively. This did not allow foradequate implementation prior to the onset of cold weatherin the Los Alamos area.

The root cause of these occurrences was determined to be amanagement problem— policy not defined, disseminated orenforced— since LANL did not have a freeze protectionpreventative maintenance standard to identify freezeprotection measures. On October 22, 1997, FSS-9 issued aFreeze Protection Maintenance and Operations Standard(MO 3.7-600). This standard requires each FMU to preparea freeze protection plan in accordance with DOEMaintenance Management program guideline, DOE Order4330.4B, section 19.1. The M&O standard section 7.8 has arequirement to “Functionally test systems controlled bydampers.” In September 1997, FSS-9 distributed a monthlynews bulletin, “FSS-9 Maintenance Talk,” to all facilitymanagers and area coordinators throughout LANLproviding basic freeze protection recommendations. Thisnewsletter specifically referred to protecting coils, citingthat heating/cooling coils in duct systems have a strongvulnerability to freeze damage. It stated that damper

Subfreezing temperaturesresulted in a series of freezeprotection events on December16-20, 1996.

The root cause of theoccurrences was determined tobe a management problem.

16

actuators should be tested. The FM for TA-35, Building 27as well as the Area Coordinator were on the distribution listfor this newsletter. Further, there has been no Laboratoryfollow-up to ensure that these lessons learned/goodpractices are implemented at an individual facility.

Other reminders have been issued with respect to freezeprotection. The LANL Operating Experience WeeklySummary, Issue 97-19, September 6-19, 1997, included anarticle on freeze protection called “Fall Means FreezeProtection.” The article refers to the fact that since 1990,LANL has experienced 31 reportable freeze protectionincidents. The DOE “Operating Experience WeeklySummary”, Issue 97-42, October 10-16, 1997, contained anarticle on Freeze Protection Reminder and Severe WeatherPlanning. In addition to the regular distribution for thisdocument, the AL Maintenance Program Managerforwarded the article to AL sites/facilities on November 4,1997, to remind them to review or establish their FreezeProtection Programs.

2.4.4 Chilled Water and Sump System Condition andMaintenance

A Condition Assessment Survey (CAS) inspection wasperformed at TA-35, Building 27 on June 6, 1997. Thereport revealed that CWE-1 failed mechanically with thefollowing deficiencies: “worn out, cabinet leaking, and thespray headers 70% plugged.” A work ticket was issued torepair these deficiencies before the formal results of theinspection were issued to the FMU; the work wascompleted in August 1997. The sump pumps, PS-1 andPS-2, were listed as poor. Poor and fail grades under theCAS indicate a need for imminent or immediatereplacement. CWE-1 was slated for replacement. At thetime of the incident, some replacement parts such as heatexchangers were on-site. However, there was no evidenceof any formal replacement schedule.

An equipment history report revealed that the followingwork was performed on the equipment involved in thisincident. CWE-1 had preventative maintenance forlubrication on April 16, 1997 and October 20, 1997. Nohistory was listed for the replacement of the spray heads inAugust 1997; however, the Area Coordinator and pipefitters verified this was performed. PS-1 and PS-2 were

On June 6, 1997, an inspectionrevealed that CWE-1 failedmechanically.

On June 6, 1997, an inspectionrevealed that the sump pumpswere in poor condition.

17

lubricated on April 16, 1997, and PS-1 was additionallylubricated on October 20, 1997.

The Area Coordinator provided to the Board a copy of a“PM checklist all facilities” dated December 2, 1997, whichretrospectively captures the normal checks within FMU-75.With respect to the systems associated in this event, thefollowing checks are pertinent:

CWE-1 Unit

• Check level of water treatment chemical barrels daily• Check water treatment concentrations and adjust as

necessary - weekly

Building 27 Mechanical Room

• Walk-through building mechanical rooms - weekly• Conduct the following every Spring and Fall:

∗ Check thermostat operation and repair, calibrateand replace as necessary

∗ Check temperature controller operations andrepair, calibrate and replace as necessary

∗ Check boiler system low limit controls, check andcalibrate in the Fall only

∗ Check unit heaters in the Fall

2.4.5 Radiological Source Control

After the flood it was difficult to determine the preciseinventory of sources on the sub-basement of Building 27.During the inventory process, NIS-5 found additionalsources that had not been on the original inventory. Thisinventory and additional inventory checks took time andmade it difficult to establish the 100% source inventoryearly in the response to the flood.

Sources stored in the sub-basement “cage” were placed onshelves and were washed away by the flooding. Moreover,one or several of the sources were physically damaged whenboxes and other material were moved in the cage during theflood clean-up. This problem applied only to small sources.Larger sources were effectively stored in polyethyleneshielded containers and Type B containers.

After the flood it was difficultto determine the preciseinventory of radiologicalsources on the sub-basement ofBuilding 27.

18

NIS-5 has accumulated a large source and materialinventory. Much of this inventory is in poor condition and isno longer used. Prior to the flood, NIS-5 initiated aprogram to locate, inventory, and dispose of all the legacymaterial that was either unused or in poor condition. A largequantity of this legacy waste material had been removedfrom the sub-basement of Building 27.

The waste removal process had proceeded significantlytoward completion. At the time of the flood, the entireensemble of NIS-5 legacy waste (sources and radioactivematerial) had been identified, inventoried, and staged on thefirst floor of Building 27. Some 700 items were identified.

2.4.6 Surveillances

Remote

Institutionally LANL has a system to remotely monitor thefunctionality of key pieces of equipment. It is at thediscretion of the FM to determine what equipment is placedon this system. Currently there is no guidancerecommending what should be placed on this system.However, at other facilities, sumps are remotely monitoredto give an indication of high level alarms. At Building 27,facility personnel stated that the only equipment they haveon the monitoring system is their boilers.

Physical

The FMU conducts regular walkthroughs of the facilities;however, there are no established criteria for conducting thewalkthroughs as well as checking operating parameters. TheArea Coordinator did develop a listing after the incident ofnormal checks within the FMU. These walkthroughs are notdocumented.

FSS-9 Facility Engineering Services

The role of FSS-9 is to provide institutional guidance andmeasure FM performance with respect to facilitymanagement, maintenance management, and facilityengineering. An additional role is to provide centralizedservices such as configuration management, planning, andrecord storage. In 1994, the FMUs were created and in1995 the FMs were given the responsibility and funding toperform maintenance for the facility. Prior to this, FSS-9coordinated and was responsible for maintenance. The

A large quantity of this legacywaste material had beenremoved from the sub-basement of Building 27.

LANL has a system thatremotely monitors key piecesof equipment, however, it wasnot being used in Building 27

FMU conducts regularwalkthrough, however, nocriteria have been establishedfor conducting them.

FSS-9 is responsible forproviding institutionalguidance and review FMperformance.

19

M&O Standards were developed from the knowledge ofJCNNM and FSS-9 to provide FMs with procedures toperform maintenance. The M&O Standards are notrequirements. However, FSS-9 is in the process ofdeveloping a LIR that will require FMs to follow certainelements of the M&O Standards that are supported by thework smart standards. FSS-9 gathers data to reflect theperformance of FMs; however, the performance measuresdo not accurately depict performance. The DivisionDirector is responsible to hold the FM accountable forfollowing the requirements. FSS-9 has identified that FMUsneed a better ways of ensuring maintenance is performed,and FSS-9 needs a better way to measure performance.

DOE

DOE LAAO has assigned FR coverage to TA-35; however,the FR who is assigned to the area is not yet qualified. Theindividual is currently assigned to completing hisqualification card for phase one to become provisionallyqualified to perform FR duties. The FR does performperiodic walkthroughs of TA-35 for orientation as well as toaccomplish qualification tasks.

DOE LAAO management has indicated that due to staffingchanges oversight of the maintenance management programhas decreased. Formerly, the FR for TA-35 and a supportcontractor provided coverage for institutional oversight ofthe maintenance management program. Currently, LAAOhas a support contractor to cover oversight of maintenancemanagement programs at LANL. The current individual isnewly assigned (less than six months) and is not cleared tovisit all facilities at LANL. DOE AL has one person, theprogram manager, to oversee maintenance programsthroughout the AL complex.

DOE LAAO has Standing Instructions defining theresponsibilities of the Facility Representatives. Semiannuallythe Occurrence Reporting Program is reviewed to appraisethe facility’s programs for investigating abnormal events. Inaddition the FRs have the responsibility to follow up andreview occurrence reports. The FR reviews and assessesoccurrence reports for facilities under his cognizance. Thefinal occurrence report is reviewed to verify accuracy of thefactual information, validation of root causes, adequacy ofcorrective actions, timeliness of scheduling the corrective

Due to LAAO staffingreductions, DOE maintenancemanagement programoversight has decreased.

20

actions, and adequacy of lessons learned. The FR alsoensures that lessons learned and generic or programmaticimplications are identified and elevated to the Assistant AreaManager for Facility Operations, as well as ensuring thatcontractor actions are taken to prevent recurrence.However, the Standing Instructions do not address howinstitutional occurrence reports are to be handled throughthis process to ensure that corrective actions are performedand are effective in preventing recurrence.

2.5 ACCIDENT ANALYSIS

An investigation of the operating conditions of the keycomponents of CWE-001 was conducted on December 1,1997. It was found that the key components were all inworking condition.

It was found, however, that the activation spring in thepressure switch associated with pump PCW-11 was broken,allowing flow to the emergency “feed and bleed” systemthrough valve V-1. Further analysis of the pressure switchfound that the setpoints were 22 psi and 30 psi (i.e., ifpressure on the outlet side of the pump dropped below 22psi, V-1 would open; when the pressure was re-establishedabove 30 psi, V-1 would close). Subsequent inspection ofthe pressure switches noted that the activation spring for thePCW-12 pressure switch was also broken. However,PCW-12 was not operational at the time of the event.

An operational test of PCW-11 showed an outlet pressureof 22 psi. It is postulated that that since the operatingpressure of this pump was right at the pressure switchsetpoint, any pressure fluctuation in drawing water fromTCW-1 would activate the pressure switch, thus openingV-1. If the system operated correctly, the resulting surge ofpressure coming off the industrial water line, whichnormally operates at approximately 100 psi, wouldbackflow toward PCW-11, thus closing the check valvelocated near the pump’s outlet. In this situation the wateraround the coils would become stagnated.

Only when the pump would start “deadheading” against theclosed check valve would the pressure increase enough tosurpass the 30-psi reset point, thus closing V-1 andreturning the system to normal operation. The deadheadpressure of the outlet side of PCW-11 as tested was 90 psi.

The pressure switches werebroken allowing the emergency“feed and bleed” system tooperate.

Water was stagnant in thechiller coils.

21

In the case of the incident, it is postulated that theoperational outlet pressure dipped under 22 psi, thusopening V-1. The resulting surge caused the pump to“deadhead,” thus increasing pressure in the outlet due to theclosure of the check valve downstream of the pressureswitch. However, the pressure switch never reset becausethe spring activator had broken. Therefore, the pumpcontinued to run and V-1 remained opened with the systemcoming into a pressure equilibrium within the coils. Thisequilibrium essentially stagnated the water in the coils whilethe brunt of the industrial water pressure was being ventedout of TCW-1’s overflow pipe (path of least resistance).

The temperature sensor measures water basin temperature.The dampers did not close because the temperature sensorwas set at 32 degrees Fahrenheit. The reason that thedampers were closed during the initial inspection of CWE-1was because of the power being shut off to the building. Atest was conducted during the Board’s investigation todetermine if the dampers operated properly. When thepower was restored to the unit, the dampers openedbecause of the 32 degree Fahrenheit setting. When thepower was shut off again, the dampers closed. Eventhoughone vane of the damper outlet was found broken, this hadno bearing on this accident. The temperature sensormeasures the temperature of the coil spray system reservoirat the bottom of CWE-1. Given that the reservoir is out ofthe direct inlet path of the cold air, receives makeup waterfrom the industrial water system, and is constantly beingrecirculated, it is unlikely the temperature would drop below32 degrees. With the 3-degree operating band for thetemperature controller, a temperature below 30 degreeswould have to be achieved to close the dampers. Therefore,the coils were exposed to the outside air at a temperature ofapproximately 11 degrees Fahrenheit during the nights ofNovember 14 and 15, 1997. The freezing of the stagnatedwater caused the coils to rupture.

Once the breakage occurred, the pressure from bothPCW-11 and from the industrial water system was relieved,thus explaining why the TCW-1 overflow only flowed for ashort time and why PCW-11 did not experience any thermaldamage due to cavitation and/or deadheading. The resultingwater flow easily surpassed the capacity of the floor drainlocated near the unit. Additionally, the mechanical roomfloor was sloped toward the stairwell to the bottom floor.

The dampers did not closebecause the temperaturesensor was set at 32 degreesFahrenheit.

The resulting water flow easilysurpassed the capacity of thefloor drain located near theunit.

22

The sump system on the bottom floor has two sump pumps,PS-1 and PS-2, which alternate in a first-stage configurationand then simultaneously if water reaches a second stagepoint as determined by a float in the sump pit. As a result ofthe sub-basement flooding, the water continued to rise inthe pit bringing the float to the second stage. However, thesecond pump was never activated because the shroudprotecting the float activated switches was slightly skewedat an angle and bound the switch activator before it couldreach the second stage (Figure 12). The flooding was morethan one pump could handle. The single pump configurationcould handle approximately 80 gallons/minute; the flowduring the event is estimated at 200 - 250 gallons/minute.The pumps continued to switch on and off but never cameon simultaneously due to the bound actuator. The two-pump configuration could handle approximately 220gallons/minute as demonstrated by post-incident testing.Eventually the water reached the electrical controllers of thesump pumps, which caused them to shut down. Thisultimately lead to the 58½ inches of water found onNovember 17, 1997.

2.6 BARRIER ANALYSIS

The Board identified numerous barriers between the chilledwater system and sump pumps and the initiating freezingtemperatures. These barriers included physical barriers,management barriers, and administrative barriers, which arepresented in tabular form in Table 1. Appendix B providesthe details of this analysis.

The physical barriers that failed or were not in place werethe dampers, sump pumps and notification system. Sincethe temperature sensor was set at 32 degrees, the dampersremained open allowing the chiller tubes to freeze andresulting in the water flowing to the sub-basement level.The sub-basement filled up with water because the secondstage of the sump pumps did not operate. The activatorarm for the second stage was bound on the shroud. Sincethere was no notification alarm in place, management wasnot aware of the 1st stage sump activation and of theflooding condition.

The management barriers that failed or were not in placewere an institutional lessons learned program, roles andresponsibilities, maintenance categorization, and oversight.

The second pump was neveractivated because the shroudprotecting the float activatedswitches was slightly skewed atan angle and bound the switchactivator before it could reachthe second stage.

Figure 12: Actuator arm for sump pumps

23

There were eight freeze protection incidents which occurredin the previous year; however, the corrective actions weredelayed until late fall. One corrective action was to developa freeze protection standard; however, LANL and LAAOdid not ensure that this freeze protection standard was in-place institutionally.

Although the FMU walked through the facility, no one wasspecifically assigned to look at the operations of theinvolved mechanical systems. There were operationalparameters for components of the chilled water system, butthe FMU did not ensure that the equipment was operatingwithin these parameters. In fact, the switches to the pumpswere broken and resulted in the stagnant water conditionand the resultant tube freeze.

Maintenance on this equipment was based on an establishedcategorization basis. One criteria for the categorization isto determine the worst case scenario. But the scenario offlooding was not used in the categorization of equipment toensure increased, formal maintenance. As a resultmaintenance activities did not prevent equipmentbreakdown. In addition, LANL and DOE did not provideoversight over the maintenance program to ensurecompliance with its requirements and guidance.

The administrative barriers that failed or were not in placewere the operations knowledge and procedures. Withoutthe operations knowledge, proper maintenance, testing andsurveillances were not performed effectively. In fact, theincorrect setting of the temperature sensor was not foundduring the walk walkthroughs by the FMU. The incorrectsetting allowed the dampers to remain open exposing thetube bundle to freezing temperatures. Also, there were noprocedures established for freeze protection plans or formonitoring of equipment, such as sump pumps. Thesenecessary documents were only issued as guidance to beused at the discretion of the FMU. Implementation of afreeze protection plan and monitoring of critical equipmentcould have mitigated, if not prevented, this incident.

PhysicalBarrier

Administrative Barriers

ManagementBarrier

Table-1 Barrier Analysis Summary

Notification Alarm

Damper

Sump Pumps

Freezing Temperature

Chilled Water System and Sump Pumps

Operations Knowledge

Procedures

24

Oversight

Categorization

Roles and Responsibilities

Lessons Learned

25

2.7 CAUSAL FACTORS

The direct cause of the accident was the improper setting ofthe reservoir temperature sensor, which caused the dampersto remain open. Thus, the cold air blew over the stagnantwater in the chiller tubes. In addition to this direct cause,there were root causes (the fundamental causes that, ifcorrected, would prevent recurrence of this and similarincidents) and contributing causes (causes that, would not,by themselves, have prevented the accident but areimportant enough to be recognized as needing correctiveaction). An Event and Causal Factors Chart is depicted inFigure 13, and a tabular summary is presented in Table 2.

Root causes of the incident were:

• Lessons Learned Program does not ensureimplementation institutionally and in a timely manner.

• The FMU was not knowledgeable of the operations ofthe mechanical system, therefore, surveillances were noteffective.

• The roles and responsibilities for the FMU with respectto maintenance were not clear or understood.

• Oversight by LANL was not performed.

• Guidance rather than procedures and standards wereprovided to the FMU.

Contributing causes of the incident were:

• Maintenance categorization of the equipment wasincorrect.

• Maintenance was not conducted on a more frequentbasis.

• Legacy design features were not evaluated after thefacility and mission changed.

• Oversight by DOE was not performed.

• Radiological source control program was notcompletely developed and implemented. It’s recognizedthat the source control program is not a contributing

Roles andresponsibilitieswere not clear

LANL andDOE oversight

was notperformed

Freezeprotectionstandardissued asguidance

LANL andLAAO

institutionallessons

learned LTA

LANL andLAAO

institutionallessons

learned LTA

Freezeprotectionstandardissued asguidance

Operationalknowledge

was noteffective

Correctiveaction is

facility specific

Correctiveactions werenot followed


was noteffective

TA-35 did notfunctionallytest louvers

Systemsetpoints not

known

Correctiveaction: raisesetpoint to

close louvers

Sump systemcategorizationwas incorrect

Correctiveactions not

timely(completed in

Fall 1997)

Increasedmaintenance

is notperformed

TA-35 did notdevelop a

freezeprotection plan


was noteffective


was noteffective

Note onequipment "setat 55 degrees"

Heatingsystem failed

at TA-48

BIO does notaddressflooding

3 correctiveactions were

developed

CWE-1 is wornout and

cabinet leaking

The Standardis not

considered"required"

Systemsetpoints not

known

Legacy designfeatures notevaluated

Systemsetpoints not

known

Sensor set at32 degreesFarenheit

Outsidetemperature 11

degreesFarenheit

Occurence reportconcerning freezeprotection at TA-48is issued 1/12/96

BIO is approved byFM 6/19/96

5000.3B "roll up"OR is issued forseries of freeze

protection events12/23/96

CAS inspection ofCWE-1 is

conducted 6/6/97

Freeze protectionstandard is issued

10/22/97

Pump (PCW-11)outlet pressure

fails belowsetpoint

Pressure switchactivates thus

releasing industrialwater

Flow throughchiller stops

Pressure switchfails to reset Louvers are open

Water freezes inbundle

11/14-16/97

Procedures notestablished for

monitoringsump pumps

Sump notmonitored

Rad sourcecontrol was not

complete

Roles andresponsibilitieswere not clear

Sump systemcategorizationwas incorrect

No screensinitially on floor

drains

Operabilitysurveillance

was noteffective

Sumpsidentified as

M4

Storage of radsources LTA

Actuator armsticks onshroud

Flooding notaddressed in

BIO

Difficulty inaccounting forsealed sources

Tubebundlebreaks

Water flowsdownstairs

First stage ofsump system

activates

Second stage ofsump system fails

to activate

Water rises insub-basement to a

level of 58.5"

PTLA arrives andsees water in

basement11/17/97 at 6:52

a.m.

Extensivepropertydamage

Pumpingoperations begin

11/19/97

Contaminationfound 11/24/97

Electricity restoredto TA-35-27

11/26/97

26

cause to the rupture of the coils in CWE-1, however,this impacted the ability to respond to this incident.

Table 2 Summary of Events and Causal Factors

Root Cause DiscussionLessons Learned Program does not ensureimplementation institutionally and in atimely manner.

In 1996, there were two OccurrenceReports, regarding nine separate incidents,related to freeze protection. In these reportscorrective actions were established toprevent other freezing incidents. Thesecorrective actions included developing afreeze protection program and raising thesetpoint for dampers. The freeze protectionplan was inadvertently issued as a guidancedocument. However, there was no follow-up by a responsible organization, to ensureimplementation of the freeze protectionplan. The corrective action also was notcompleted until late October 1997 whenfreezing conditions could occur. Correctiveactions were not effective in preventingrecurrence.

The FMU was not knowledgeable of theoperations of the mechanical system,therefore surveillances were not effective.

Since the operational setpoints of thesystem were not known, the FMU teamcould not ensure that the systems operatedproperly when they conducted theirsurveillance or walkthroughs. Although theoperational setpoint for the CWE-1temperature sensor was written on theequipment, neither the FMU team membersnor the maintenance personnel wereknowledgeable of the parameters. Inaddition, operation of the pumps andpressure switches were not understood.

The roles and responsibilities for the FMUmaintenance were not clear or understood..

The FMU did not establish clear anddefinitive roles and responsibilities.Therefore, no individual was responsible forchecking operational performance and setpoints of the systems.

Oversight by LANL was not performed. Although roles and responsibilities wereestablished in various LIR and standards, noorganization ensured that the FMU was

27

Root Cause Discussionmeeting the established roles andresponsibilities and performance objectives.

Guidance rather than procedures andstandards were provided to the FM.

The freeze protection M&O standard wasissued as guidance. In addition, only aguidance document exists for categorizingequipment and maintenance. Since thesestandards are optional, effective andconsistent maintenance activities are notperformed. Had the FMU developed afreeze protection plan in accordance withthe guidance provided, the dampers shouldhave been functionally tested leading toresetting the temperature sensor from 32°Fto an appropriate setting. There is noinstitutional procedure to require remotesurveillance of important equipment such asthe sump pumps. Although the secondstage on the sump pump failed because theactuator arm was bound by the shroud, asump alarm would have alerted theappropriate personnel of the condition andresponse would have limited the extent ofthe property loss.

Contributing Cause DiscussionCategorization of the equipment wasincorrect.

The sump pumps were identified as an M4system. This categorization was incorrectsince the basement area stored numeroussealed radiological sources vulnerable towater damage and the basement housedover three million dollars in assets.

Maintenance was not conducted on a morefrequent basis.

Maintenance on equipment was minimal.Increased maintenance was not conductedbased on history, potential failure andimportance to mission.

Legacy design features were not evaluatedafter the facility and mission changes.

The initial design of the facility included a2½ inch backup “feed and bleed” coolingsystem. When the chiller tubes failed, the200 gpm flow overcame the first stage ofthe sump pump. The redesign or removal ofthis system would have reduced the amountof flood damage. DOE oversight was notperformed.

28

Root Cause DiscussionDOE oversight was not performed. The reduction in staff and qualified

individuals resulted in inadequate oversightof their institutional managementmaintenance program.

Radiological source control program wasnot completed. It’s recognized that thesource control program is not acontributing cause to the rupture of thecoils in CWE-1, however, this impacted theability to respond to this incident.

Three inventories had to be performed toensure 100% accountability of all sources.The sources were stored on shelves andwere washed away during the flooding andcould have entered unscreened drains withinthe caged area. The sources becamephysically damaged as a result of theflooding and clean-up activities resulting incontamination. Some actions had beentaken by NIS-5 prior to the flood; however,had actions been taken more aggressivelysome of the difficulties in response to theflood would have been avoided orminimized.

3.0 Conclusions and Judgments of Need

This section of the report identifies the conclusions andjudgments of need determined by the Board as a result ofusing the accident analysis methods described in Section1.4. Conclusions of the Board are those consideredsignificant and are based upon facts and pertinent analyticalresults. Judgments of need are managerial controls andmaintenance standards believed by the Board to benecessary to prevent or mitigate the possibility or severity ofa recurrence of this type incident. Judgments of need flowfrom the conclusions and causal factors and are directed atguiding managers in developing follow-up actions. Table 3lists the conclusions and the corresponding judgments ofneed identified by the Board.

Table 3 Conclusions and Judgments of Need

Conclusions Judgments of NeedApplication of lessons learned fromfreeze protection incidents was noteffective.

LANL needs to ensure that institutional correctiveactions are uniformly and promptly implemented.

LAAO needs to establish a process to ensure importantinstitutional corrective actions, based on lessonslearned, are implemented at LANL.

29

FMU-75 did not develop a freezeprotection plan and LANL did notensure that a freeze protection planwas developed.

LANL needs to establish requirements for FMUs todevelop freeze protection plans and ensure they areimplemented at all FMUs.

FMU-75 did not take actions toprotect TA-35, Building 27 sub-basement from a flood.

LANL needs to re-evaluate equipment categorizationand “legacy design” for failure modes to preventpotential flooding.

LANL needs to ensure operability of all safetyequipment including sump pumps in the basement andsub-basement.

NIS-5 did not complete plannedradiological source control actionswhich would have facilitated floodresponse and minimized flooddamage.

NIS-5 needs to complete radiological source controland storage actions.

Institutional maintenancemanagement program lacks clearrequirements, training, and oversight.

LANL needs to evaluate current requirements orestablish requirements for the categorization,maintenance, and surveillance of systems. Clear rolesand responsibilities and associated training to provideunderstanding of both responsibilities and operationsneed to be developed. Evaluation of performancethrough oversight to these requirements, needs to beperformed to provide institutional consistency andincorporation of shared corporate knowledge betweenFMUs. All FMs need to be held accountable for theoperation of the facility.

The reduction in staff and qualifiedindividuals resulted in inadequateoversight of the institutionalmanagement maintenance program.

DOE/AL and LAAO need to strategically alignresources in order to provide oversight of themaintenance management program.

APPENDIX A

APPENDIX B

Appendix B Barrier Analysis

Barrier Type Hazard Direct Barrier orControl Failure

Possible Contributing Factor toBarrier or Control Failure

Possible RootCauses of Failure

Loss Evaluation

Physical Barrier Chilled WaterTubes Freeze

Properly runningtransport equipment

Setpoints of pressure switches werenot verified nor known; pumpperformance deteriorated to low setpoint.

Aging anddeterioration

CWE-001 With aging and deterioratingequipment, increased maintenance andsurveillance is needed. The CAS reportfor Building 27 had identified thedeteriorating condition of the CWE-001.


Emergency feed andbleed system from theindustrial water system

Emergency system valve could notbe shut due to broken pressureswitches. Industrial water equalizedthe pressure in the coil systemresulting in no flow through thetubes.

Lack ofmaintenance,inadequateoperationalsurveillance; andlegacy design notsuited for currentoperations

CWE-001 The FMU was not aware of the systemoperation and its setpoints; therefore,appropriate maintenance andsurveillance activities were notestablished. The building was designedfor a nuclear reactor and was notchanged to accommodate the currenttenant. The current facility housesoffices, labs and training operations.The coil tubes did not have anypressure/drain valves, which resultedin the stagnation of water.


Backup pump andpressure switch

The other pressure switch wasbroken and there was no automaticswitching gear for backup pump.

Lack ofmaintenance,inadequateoperationalsurveillance, andinadequate design

CWE-001 The FMU was not aware of the systemoperation and its setpoints; therefore,appropriate maintenance andsurveillance activities were notestablished


Temperature sensor System had no freeze stat on theinlet air.

Legacy design notsuited for currentoperations

CWE-001 Without a freeze stat, there wasreliance on the basin temperaturesensor and operational surveillances.


Dampers Dampers remained open due to the32°F sensor setting allowing 11°Fair to blow over the coils.

Unauthorized/inadvertent changeof temperaturesetting.

CWE-001 The temperature sensor was notsecured from individuals makingunauthorized or inadvertent changes.




Loss Evaluation


Wall thickness ofcopper coils

Expansion of water breaks tubing. Pressure buildup byice exceeds designcapacity of thecopper coils

CWE-001 There was an initial concern that thetubes were also deteriorated but it wasfound that the tubes were replacedthree years earlier.

Physical Barrier BasementFlooding

Sump system 1st Stage, one pump operation isovercome and 2nd stage, two pumpoperation does not activate due toactivator arm bound on floatshroud.

Inadequateoperationalsurveillance andinadequate design

Waterdamage

Better method was needed to secureshroud rather than using one hoseclamp. Also, the sump was identifiedas M4 rather M3, which would haverequired a more rigorous approached tomaintenance.

Physical Barrier BasementFlooding

Notification alarm Neither the sump nor theoperability of the CWE-01 isremotely monitored.

Inadequate designand ineffectivepolicy andprocedures

Waterdamage

Installation of a notification alarmwould have mitigated the incidentsince the incident occurred over theweekend. However, there are norequirements established to requirenotification alarms although there wasa high property value located in thesub-basement.

ManagementBarrier

Chilled WaterTubes Freezeand BasementFlooding

Direct supervision ofmaintenance

No one makes effective operationalsurveillances on the equipment.

Roles andresponsibilities notclear and lack ofoperationalsurveillance

CWE-001and waterdamage

The roles and responsibilities wereunclear; as a result, all members of theFMU walked through the facility butno one specifically was assigned tolook at the operations of the systembased on the institutional guidance.The FMU was not fully knowledgeableof the operating parameters of thesystems and therefore could notperform adequate surveillances.Although the FMU conductedwalkthroughs of the equipment areas,they did not notice the improper settingof the temperature sensor or the floatarm being bound by the shroud.

ManagementBarrier

Chilled WaterTubes Freezeand Basement

Categorization ofequipment based onrisk assessment

The equipment was categorized M-4 rather than M-3.

Policies andprocedures notclearly defined


In determining the categorization perthe institutional guidance, the worst-case failure mode, the operating history




Loss Evaluation

Flooding and the insufficient maintenance werenot considered; therefore, anunanalyzed event (flooding) occurred.

ManagementBarrier


Conduct ofmaintenance

FSS-9 establishes policy andprocedures.

No oversight toensure compliancewith policies andprocedures


There is no evaluation of the FMperformance to the institutionalmaintenance guidance; therefore, theFMs are not accountable for propermaintenance.

ManagementBarrier


Lessons learned Institutional freeze protectioncorrective actions were noteffective.

Lessons learnedineffective


Although required to prepare a freezeprotection plan and functionally testdampers, there was no follow-up todetermine effective implementation ofthese corrective actions. Also, theseasonal timeliness of the correctiveactions caused delay. The correctiveactions for institutional lessons learnedare not tracked to closure by the FR asare facility corrective actions. As aresult, LAAO did not ensure that thefreeze protection plan was institutedLANL-wide.

ManagementBarrier


FM oversight FMs operate without any feedbackto performance

No oversight toensure compliancewith policies andprocedures


Without accountability, FMs are free tooperate independently andinconsistently.

ManagementBarrier


DOE oversight ofMaintenanceManagement

DOE/AL and LAAO staffingchanges has reduced maintenancemanagement oversight as well asthe number of qualified individualsperforming oversight.

Reduced DOEoversight to ensurecompliance withpolicies andprocedures


The reduction in staff and qualifiedindividuals resulted in inadequateoversight of their institutionalmaintenance management program.

AdministrativeBarrier


Procedures/instructions

No FMU directed procedures; FSSprocedures are established asguidance.

Inadequate/inconsistentmaintenance


If there are only guidance documents,there is little consistency in the conductof maintenance across the Laboratory.


Chilled WaterTubes Freeze

Maintenance History Maintenance history is notdocumented and used for decision-

Inadequate/ineffective

CWE-001and water

If the maintenance history showedincreased maintenance and




Loss Evaluation

and BasementFlooding

making or maintenance strategy. maintenance damage deterioration, then decisions could bemade to replace or monitor morefrequently the affected equipment.



Operator systemknowledge

Equipment was not operated withinthe design parameters.

Inadequateoperations andmaintenancestrategy


Without the operations knowledge,proper maintenance, testing, andsurveillances can not be performedeffectively. Also, if the FMU had beenaware of the important thermostatsetting for the dampers, the thermostatcould have been secured or at leastmonitored on a frequent basis to ensurethe 55°F set point.



Training Roles and responsibilities as well assystem operability were notunderstood by the FMU.

Lack of formaltraining program


With adequate training, the FMU couldhave developed necessary maintenanceand operations procedures, especiallyin the area of operating conditionsurveillances. Inadequate andineffective training resulted in a lack ofmaintenance and operationsunderstanding.

Documents

Type B Accident Investigation Board Report Chiller Line Rupture … · 2014-04-17 · Accident Investigation Board (Board) ... knowledgeable of the operations of the mechanical systems,