Data centre design · 2017-03-14 · room and its support areas,” according to TIA 942. This design guide is based upon the requirements of TIA 942 Telecommunications Infrastructure

© 2006 Connectix Ltd

Contents

1.0 Purpose

2.0 Disclaimer

3.0 Introduction

4.0 Physical location of the data centre

5.0 Sizing and capability audit

6.0 The hot aisle/cold aisle design concept

7.0 Specifying a raised floor

8.0 Equipment racks and cabinets

9.0 Heating, Ventilation and Air Conditioning (HVAC) within the data centre

10.0 Electrical systems to and within the data centre

11.0 Earthing, bonding and the Signal Reference Grid

12.0 Fire detection, alarm and suppression within the data centre

13.0 Communications cabling and containment

14.0 Security, access control and CCTV

15.0 Building Management Systems, from rack to room level

16.0 Tiering, H&S and other project management issues

Appendix 1 Standards referenced

Consultant ProgrammeWHITE PAPER

FEBRUARY 2006

Data centre design

Document ref: EE-TDT-05-002 ENGINEERING EDUCATION Ltd. ©Issue 001 License EEL05L02

By Barry Elliott

BSc, MBA, RCDD,

C.Eng, MIEE

barry.elliott@

engineeringeducation.co.uk

© Engineering Education

Ltd 2006

Registered office

22 Foxcover Road

Heswall Hills

Wirral CH60 1YB

UK Head Office:

Connectix Limited

33 Broomhills Industrial Estate

Braintree

Essex

CM7 2RW

Tel: 01376 346 600

Email: [email protected]

Republic of Ireland Office:

Connectix Limited

29 Westlink Industrial Estate

Kylemore Road

Dublin 10

Ireland

Tel: 01623 6751

Email: [email protected]

February 2006 © 2006 Connectix Ltd

Data centre design

1.0

2.0

Purpose

This document is a design tool to assist designers to identify all the processes and activitiesrequired to fully define the requirements of a data centre to industry standards and bestpractice parameters. It will allow a preliminary design stage to be reached, with a clientfeedback loop, enabling full costed design proposals to be undertaken.

Disclaimer

This document is intended for the use of persons qualified in the electrical, mechanical andconstruction requirements of a data centre. This document quotes figures and extracts frominternational standards but this does not absolve the user from full knowledge and usage ofthe original standards themselves. Every effort has been made to supply a complete and up-to-date technical précis of the current international, European and British standards andregulations concerned but the fitness-for-purpose and final design remains the responsibility ofthe document user.

Except where other documents have been quoted, this document remains the copyright ofEngineering Education Ltd and its reproduction is forbidden under the Copyright, Designs andPatents Act 1988. Licences may be obtained from [email protected].

Introduction

A data centre is;

“A building or portion of a building whose primary function is to house a computer room and its support areas,” according to TIA 942.

This design guide is based upon the requirements of TIA 942 TelecommunicationsInfrastructure Standard for Data Centers, April 2005.

Although this is an American standard invoking other American standards and codes it is farmore substantive than the equivalent CENELEC EN 50173-5 Data centre standard, which isstill at draft stage. However this document expands upon the TIA 942 standard andincorporates all the requirements of European and British standards, Directives andRegulations. These include EN 50173, EN 50174, EN 50310, BS 5839, BS 6701, BS 7671, theUK Building Regulations, the Disability Discrimination Act and many others. They are alldetailed in Appendix 1.

Many diverse areas need to be addressed to fully design and specify a data centre. It isessential that it is agreed at the start of the project exactly who is responsible for every item orelse the final build will be severely compromised if a vital design element has been overlookedor is incompatible with other services.

3.0

January 2006 © 2006 Connectix Ltd

Data centre design

A data centre design project can be split into the following sections;

1. Location2. Construction3. Definition of the spaces and size available4. Planning the layout of the computer room floor5. Designing the raised floor6. Calculating day one and future IT requirements7. Calculating the day one and future air conditioning requirements8. Deciding upon the type and location of the air conditioning units9. Calculating day one and future power supply requirements10. Sizing and location of UPS and standby generators11. Designing the earth bonding and signal reference grid12. Designing the power distribution system within the computer room and within the

equipment racks13. Lighting, emergency lighting and signage14. Access control, security and CCTV requirements15. Fire detection, alarm and suppression system, including hand-held fire extinguishers16. Specifying and designing the structured cabling system and its containment system17. Organising connections to external telecommunications providers and the ‘Entrance

room’18. Integration of Building Management Systems with other command and monitoring

networks and their appearance at a control room19. Project management issues, health & safety and ongoing operational and maintenance

issues

Data centre projects are either green field new-build projects or conversion/renovationprojects. In either case it is advisable to undertake a complete audit of what exists already oron the proposed designs.

Apart from meeting the day one designs and proposed expansion plans it is also necessary todecide upon which level of backup or redundancy will be built in to the finished location. Fordata centres these levels are now designated as being of Tier 1, 2, 3, or 4, with Tier 4 beingthe highest level of redundancy.

The Tiering level is described in great detail in the TIA 942 standard which in turn has takenmuch of its philosophy from the Uptime Institute. A very brief summary is given in the tablebelow. In the terminology of redundant systems ‘N’ means enough equipment to do the job,N+1 means one more additional unit to act as a redundant supply whereas 2(N+1) means twoindependent paths to complete the job.


Data centre design

Tier I Tier II Tier III Tier IVSite availability 99.671% 99.749% 99.982% 99.995%Downtime (hours/yr) 28.8 22.0 1.6 0.4Operations Center Not required Not required Required RequiredRedundancy for power,cooling

N N+1 N+1 2(N+1)

Gaseous fire suppressionsystem

Not required Not required Approvedsystem

Approvedsystem

Redundant backbonepathways

Not required Not required Required Required

The relationship of the Spaces within a Data centre


Data centre design

4.0 Physical location of the Data Centre

Physical location and architectural audit

Parameter Recommendation Ref.

4.1 Does the building and rooms exist,or are building works required?

4.2 Any known seismic problems?

4.3 Any known subsidence problems?

4.4 Any known flooding problems? Not on a 100 year flood plain TIA 942 (F.6)

4.5 Any known security/ criminal problems likely with this area?

4.6 Is connection to mains/telecoms services available?

4.7 Is there a very close proximity to main Should be 0.8km away from a TIA 942 (F.6)roads, railway lines, airports, oil or major highway and 0.4km awaychemical storage or works from chemical plants, dams etc

4.8 Is there easy access to the site?

4.9 Are their lifts/goods lifts available if not on the ground floor?

4.10 Are there any excessive external noise sources?

4.11 Will this unit be a cause of noise or disturbance to adjoining offices?

4.12 Any potential EMC problems, Any interfering fields should be TIA 942 (F.2)e.g. mobile phone masts, lift motors less than 3V/mon the other side of a wall etc?

4.13 Is there access to a suitable external site for the air con heat exchangers?

4.14 Any other known safety or locationissues that need to be recorded such as presence of asbestos?

4.15 Is the building or room susceptibleto lightning strikes?

4.16 Is out of hours access to the site possible?

Are there any issues concerningplanning permission or conservationzones or building listing?

4.17 Are there separate office, storage, or parking areas available for contractors?

4.18 Has the room or design been audited to The requirements of the Disabilitycomply with the Disability Discrimination Act may be taken Discrimination Act?* from;

BS 8300:2001 Design of buildings and their approaches to meet the needs of disabled people — Code of practice, and Building Regulations 2000 Part M Access and facilities for disabled people


Data centre design

5.0 Physical sizing and capability of the Data Centre

Sizing and room capability audit

Parameter Recommendation Ref.

5.1 What are the dimensions of the data centre?

5.2 What are the dimensions of the computer room?

5.3 What other areas have been allocatede.g. office area, entrance room etc?

5.4 What is the height of the computer room? Min of 2.6m from finished floor TIA 942

5.5 Is the floor load capacity acceptable? The minimum distributed floor TIA 942loading capacity shall be 7.2kPA. The recommended distributed floor loading capacity is 12kPA

5.6 Where are the doors and what are their Doors shall be a minimum of 1m TIA 942sizes? wide and 2.13m high, without

doorsills, hinged to open outward(code permitting) or slide side-to-side, or be removable. Doors shall befitted with locks and have either nocenter posts or removable center posts to facilitate access for large equipment. Exit requirements for the computer room shall meet the requirements of any other local requirements.

5.7 Is there lighting in place and is it adequate? Lighting shall be a minimum of TIA 942500 lux in the horizontal plane and 200 lux in the vertical plane, measured 1m above the finished floor in the middle of 4 all aisles between cabinets.

5.8 Is emergency lighting and signage Principally described in BS 5266-1, BS 5266-1fitted/planned? The Code of Practice For

Emergency lighting, amongst others.Exit signage. Principally described BS 5499-4in BS 5499-4:2000 Safety signs,including fire safety signs. Code ofpractice for escape route signing

5.9 Will the emergency lighting require its To BS 5266 BS 5266own battery back-up supply?

5.10 Is the basic décor acceptable?* Décor to be finished in a light colour TIA 942with minimal glare and dust generation

5.11 Does equipment not related to the There should be no other services TIA 942support of the computer room (e.g., passing through the computer roompiping, ductwork etc.) pass through,or enter the computer room?

5.12 Is there a fresh water supply and drainage network available?


Data centre design

6.0 The hot aisle/cold aisle concept

In trying to design a standardised, modular and upgradeable space for I.T. andcommunications equipment much thought needs to be given to rack location and the methodof supplying power, communications and refrigerated air to it.

The ‘standard’ model has been defined by TIA 942, ASHRAE and other authorative sources asbeing based on a front-to-back cooling regime based on rows of racks facing each other.Cold air is supplied to the front of these racks through air vents placed in the raised floor infront of them. The chilled air is fed to these vents from air conditioning units blowing into theplenum space formed by the raised floor. The vented aisle is thus known as the cold aisle andthe cold air is drawn through the equipment racks by the I.T. equipments’ own fans andexpelled out of the back into what is now the hot aisle. The rising hot air from this aisle findsits way back to the air conditioning unit to be chilled and then to repeat the cycle.

The fronts of the two facing racks are two whole floor tiles apart and when the depth of therack and the necessary access clearance space behind it is taken into account we can seethat the minimum realistic pitch before the process repeats itself is seven tiles.

Feeding cold air through standard 25% open floor vents into a rack with no additional coolingmethods normally limits the heat dispersion to about 2kW per rack, or about five averageservers. Other upgrade paths are available to get more air through the rack and this will beexplained in more detail later.

A lot of communications equipment is designed for side-to-side cooling and so additionalconsideration needs to be given to cope with this variation but in general the hot aisle/coldaisle, 7-tile pitch system is generally considered to be the ‘base’ model by the relevantstandards and industry sources.


Data centre design

7.0 Specifying a raised floor

The raised floor height will be based on 600 x 600mm floor tiles with an anti static finish to IEC 61000-4-2 and not less than 300mm in height. A guide to floor heights, when used as anair distribution plenum, comes from VDI 2054, Air conditioning systems for computer areas.

Other guidance for floor height comes from;• 450 – 600 mm, IBM• 450 min, 600 ‘ideal’, SUN• 300 – 600 mm, BS EN 12825• 300 mm min, TIA 569

The Property Services Agency (PSA) Method of Building Performance Specification 'PlatformFloors (Raised Access Floors)', MOB PF2 PS, became the de facto industry standard in the UKfor about 20 years until the recent arrival of the BS EN 12825:2001 specification.

In July 2001 a European Standard EN 12825 Raised access floors, was approved by CEN as avoluntary specification for private projects and mandatory for public projects.

For the floor strength the minimum distributed floor-loading capacity shall be 7.2kPA. Therecommended distributed floor loading capacity is 12kPA (TIA 942). From MOB PF2 PS andBS EN 12825 this means specifying ‘Heavy Duty’ or preferably ‘Extra Heavy Duty’ floor grade.

The plenum area formed under the raised floor must be clean, sealed, dust free, fitted with avapour barrier and sealed to a level of air permeability of at least 3m3/h/m2 at 50 Pa (BuildingRegs Part F).

The reasons for pressure sealing the plenum area are;

• Chilled air conditioned air will be able to escape through poorly finished floor tiles andservice penetrations, leading to;o More electricity consumed to replace that airo An inability to deliver the volume of chilled air required at the floor ventso A variation in air pressure across the floor leading to an inability to deliver chilled air

at the air vents• Unsealed service penetrations (cables/pipes etc) into the plenum area are a fire risk and

will allow the spread of fire and smoke into or out of the computer room (Building Regs,part B)

• Gaseous fire suppression systems rely on lowering the level of oxygen available to firesand depend upon a sealed area to work in to prevent oxygen from re-supplying the fire.BS ISO 14520 P1: 2000(E), Gaseous fire-extinguishing systems. Physical properties andsystem design. General requirements, requires a pressure test every twelve months

Floor area Height of raised floor according to VDI 2054

200 - 500m2 Approx 400mm

500 - 1000m2 Approx 700mm

1000 - 2000m2 Approx 800mm

>2000m2 >800mm


Data centre design

An aspirating (early warning) smoke detection system shall be placed in the plenum zone(TIA 942).

Where a need for a fire suppression system in a sub floor space is deemed appropriate,consideration should be given to clean agent systems as a means to accomplish thisprotection (TIA 942).

The under floor area must not be used for any other purpose other than the supply of air andthe distribution of cables. Cables must be fire rated according to the local jurisdiction andmust be placed so as not to impede airflow. All redundant cables must be removed (NationalElectrical Code 2002).


Data centre design

8.0 Equipment racks and cabinets

Computing and communications equipment has been located in racks, usually 19-inch based,for at least the last thirty years. Racks, or frames, come in all shapes and sizes; from a fewhundred millimetres high to over two metres high; 600 or 800 mm wide and from 600 to 1200mm deep. The internal fittings are usually based on a 23-inch pitch for telecommunicationsand 19-inch for everything else. A handful of EIA, IEC and ETSI standards cover the physicaldimensions of the rack, such as EIA-310-D. The vertical spacings for the installed equipmentare based on Rack Units, or just ‘U’, where one U is 44 mm.The main frame of the rack can be based on a four-post construction, i.e. to make arectangular frame, or the space-saving two-post system which is essentially two pieces ofvertically placed metal spaced 19-inches apart (apologies for mixing metric and imperial unitshere but that is the common practice!). A server rack needs to be a four-post enclosed unit.

The purpose of the rack is;• To hold and securely locate electronic equipment• To provide an organised routing for power and communications cabling• To assist in the airflow and cooling of the equipment• To provide the above in an aesthetically pleasing construction

8.1 Size Racks/cabinets are usually 600 mm wide and with a useable internal space of 42U for 19-inchrack-mounted equipment. This gives a rack height of just over two metres. Slightly larger (andof course smaller) versions are available but 42U seems a popular choice. Depth is at least800 mm but may be up to 1.2 m. A one-metre depth allowance seems average.

TIA 942 statesRefer to ANSI T1.336 for additional specifications for cabinets and racks. In addition to therequirements specified in T1.336, cabinets and racks heights up to 2.4 m and cabinetdepths up to 1.1 m may be used in data centers (although 2.1 m is recommended).

Cabinets should have adjustable front and rear rails. The rails should provide 42 or morerack units (RUs) of mounting space. Rails may optionally have markings at rack unitboundaries to simplify positioning of equipment. Active equipment and connectinghardware should be mounted on the rails on rack unit boundaries to most efficiently utilizecabinet space.

If patch panels are to be installed on the front of cabinets, the front rails should berecessed at least 100 mm (4 in) to provide room for cable management between the patchpanels and doors.


Data centre design

8.2 Ventilation This is a key area of differentiation between ‘standard’ equipment racks and server racks. Aserver rack must cope with the ventilation demands of many kilowatts worth of electricalequipment. A standard glass-fronted rack with horizontal fan tray fitted can only cope with thecooling demands of less than a kilowatt.

It would appear that a suitably ventilated rack, supplied with adequate chilled air through astandard floor tile, can cope with about two kilowatts of heat dissipation, where the motiveforce through the rack is only provided by the fans within the server units themselves.

The amount of ventilation required is stated by several sources and is expressed as a ratio of‘open’ space to overall door area, e.g.;• …..servers require that the front and back cabinet doors to be at least 63% open for

adequate airflow. SUN• One method of ensuring proper cooling is to specify a rack doors that provide over 830

in2 (0.53 m2) of ventilation area or doors that have a perforation pattern that is at least63% open. APC

• Racks (cabinets) are a critical part of the overall cooling infrastructure. HP enterprise-class cabinets provide 65 percent open ventilation using perforated front and rear doorassemblies. To support the newer high-performance equipment, glass doors must beremoved from older HP racks and from any third-party racks. HP

• …the cabinet should either have no doors or, if required for security, doors with aminimum 60% open mesh for maximum airflow and is best not equipped with topmounted fan kits. Chatsworth

• Ventilation through slots or perforations of front and rear doors to provide a minimum of50% open space. Increasing the size and area of ventilation openings can increase thelevel of ventilation. TIA 942


Data centre design

When the heat load goes above about 2 kW (about 5 average servers) then an escalationpolicy is required, which can take the form of;• Increasing floor tile vent size up to 75% open area• Replacing floor tiles with fan assisted grate tiles• Adding specialised fan units to the top and/or bottom of the rack• Using cabinets where the entire rear door is a fan unit

The above solutions will take the heat dissipation capability up to about 6 kW per rack. Abovethat then more specialised racks need to be used where the whole rack is fed by a chilledwater supply. These designs can cope with loads in excess of 20 kW. New designs usingliquid carbon dioxide claim cooling capacities of over 30 kW per rack.

It is also important that the front to back cooling scheme adopted in such racks is notcompromised by gaps in the rack allowing cooled air to mix with hot air drawn back throughthe gaps (Thermal Guidelines for Data Processing Environments –ASHRAE). For this reason allgaps in the rack must be filled in with blanking plates. Also excessive gaps for cabling at theside of the racks should be sealed with an air dam kit and any cable entry points at the bottomof the rack should also be sealed with a brush strip.

8.3 PowerThe rack needs to be powered and in Europe this would generally be provided by a 16 or 32amp, 230 V single phase feed through an IEC 60309 connector. At least two feeds arerequired for redundancy and backup purposes so a dual 32 amp feed would be counted assupplying 32 x 230 = 7.36 kVA (remember that useful power is measured in watts, which isamps x volts x power factor).

For loads above 7 kVA then either more 32 amp feeds are supplied or a three-phase supply isprovided which would normally deliver at least 22 kW through a five-pin version of the IEC60309 connector. For three-phase supply Regulation 514-10-01 of BS 7671 requires a warningnotice to be secured in such a position that the warning is seen before access is gained to liveparts.

Within the rack the power is distributed by what is widely known as a power distribution unit,or PDU. There does not seem to be a widely accepted definition of a PDU and at its simplestit is just a power strip of sockets that distributes the incoming electricity to the rackequipment. However more functionality is available in the form of;• Sequential start up• Automatic crossover switch between two supplies• Power line conditioning• Reporting function about status and power usage. This in turn may be a simple LED

readout on the unit or part of an IP addressable managed system


Data centre design

8.4 Control and monitoringA data centre server rack must be secure and be able to monitor and report its environmentalstatus back to some central control point. The monitoring system may be part of a building-wide Building Management System (BMS), an add-on localised monitoring scheme or a built inrack-monitoring scheme designed and dedicated to the task. TIA 942 states ‘A BuildingManagement System (BMS) should monitor all mechanical, electrical, and other facilitiesequipment and systems.’

The rack sensor system should be able to detect the following;• Temperature• Smoke• Water• Humidity• Access• Vibration• Airflow• Particles in the incoming airflow

And respond with one or more of the following;• Visual alarm on top of cabinet• Audible alarm• Networked alarm• CCTV

8.5 Rack locationThe ‘standard’ model described in TIA 942 and elsewhere depends upon the hot-aisle/cold-aisle concept described in section 6 of this document. In this model chilled air is pumped outinto the plenum/raised floor area beneath the racks and made available by vented floor tilesplaced in front of the racks. This also requires the 7-tile pitch approach.


Data centre design

5.0The 7-tile pitch requires that the front edges of the two facing cabinets are placed in line withthe edge of a floor tile, and two complete floor tiles, i.e. 1.2 m, separates the two facingcabinets, thus forming the cold aisle. The depth of the rack will cover about one and a halffloor tiles and so a complete floor tile is needed in the hot aisle for access. This arrangementmeans that the set will repeat itself every seven tiles, or 4.2 metres

Apart from the 7-tile arrangement TIA 942 also requires clearances of a minimum of 1 m offront clearance for installation of equipment and a minimum of 0.6 m of rear clearance forservice access at the rear although a rear clearance of 1 m (3 ft) is preferable. Some rackshave split rear doors to facilitate rear clearance.

IEEE 1100, referenced in TIA 942, suggests a clearance of two metres from building structuralsteel in case of lightning flashovers.

8.6 Cable managementCables may enter from the top or bottom or both of the rack. If coming up from the bottomthen a cable brush seal is required to prevent chilled air from entering and confusing the front-to-rear airflow scheme.

All cables shall be neatly dressed and secured with minimum bend radii protected accordingthe standards or manufacturers’ instruction. All cables must be adequately labelled asdescribed in TIA 942, TIA 606 and elsewhere.

A vertical cable manager shall be installed between each pair of racks and at both ends ofevery row of racks. The vertical cable managers shall be not less than 83 mm in width. Wheresingle racks are installed, the vertical cable managers should be at least 150 mm wide. Thecable managers should extend from the floor to the top of the racks.

Horizontal cable management panels should be installed above and below each patch panel.The preferred ratio of horizontal cable management to patch panels is 1:1.

8.7 Health and SafetyEquipment racks can be very heavy, over 500 kg. It is essential that;1) The concrete floor beneath the raised floor is strong enough, and finished flat2) The raised floor is strong enough, the pedestals are securely fixed and the floor is finished flat3) Equipment racks are leveled onto the raised floor4) Local seismic regulations for fixing are obeyed5) Heaviest equipment is placed at the bottom6) Extendible stabilizers are used when sliding heavy equipment out of a rack7) Racks are bayed together8) All racks, including doors, are earthed according to local regulations9) Any removed floor tile positions are surrounded by warning signs


Data centre design

9.0 Heating, ventilation and air conditioning

TIA 942 recommends that the following conditions be maintained in the computer room;• Relative humidity: 40 to 50 %• dry Bulb Temperature: 20°C to 25°C• Max dew point: 21°C• Max rate of change: 5°C per hour• A positive pressure will be maintained with respect to surrounding areas

The precision air conditioning facility must be available 24 hours a day, 365 days per year andconnected to the standby generator in the event of a mains failure.

The ambient temperature and humidity shall be measured after the equipment is in operation.Measurements shall be done at a distance of 1.5 m above the floor level every three to sixmetres along the center line of the cold aisles and at any location at the air intake of operatingequipment. Temperature measurements should be taken at several locations of the air intake ofany equipment with potential cooling problems. Details are contained in Thermal Guidelines forData Processing Environments.

Air conditioning may be achieved by either;• Direct expansion Computer Room Air Conditioning units (CRAC) in the computer room• Centralised chiller units supplying chilled water to heat exchange units within the

computer room• Chilled water supplied directly to heat exchange units built into equipment racks

Or any combination of the above.

Small to medium sized data centres tend to go for the direct expansion, DX, CRAC unitsplaced in the computer room. Larger facilities tend to go towards the centralized chiller andcold water distribution. Directly cooled racks have so far tended to be an upgrade path whenconventional room cooling runs out of capacity but there is no reason why they couldn’t bedesigned in from the start, especially when floor space is at a premium.

The mathematics of air conditioning shows that to remove one kilowatt of heat and cool anitem by around 11°C, approximately 160 cfm (cubic feet per minute) or 74 litres/second of airneeds to flow through that equipment.

The literature suggests that in practice an adequately constructed and sealed raised floor,supplied with adequate chilled air, can supply about 320 cfm of air through a standard 25%floor vent, which implies that one floor vent, in these circumstances, can cool around 2 kW ofequipment if placed in front of an equipment rack. There are many variable in this equation, e.g.• Are the CRAC units supplying a sufficient volume of air at the correct temperature?• Is the underfloor plenum area deep enough and clutter free to allow free airflow?• Is the underfloor plenum sealed enough to maintain the correct excess air pressure?• Is the excess pressure evenly distributed around the floor area? This in turn depends

upon the above factors, plus depth of floor void and number, size and location of otherfloor vents


Data centre design

A successful design thus depends upon;• Designing an appropriate raised floor• Sizing the air conditioning requirements in light of day-one load, future expansion and

redundancy requirements• Correctly positioning the CRAC units and air return path• Correct location of the equipment racks in the hot-aisle/cold aisle format with the 7-tile

pitch layout• Correct location of the floor vents to deliver the chilled air directly to the rack• Correct construction and loading of the rack to maintain the desired airflow• Correct location of the external air handling/chiller units, which need a strong mounting

plinth, a secure area and electrical and plumbing connections

Up until the early part of this century the average heat load developed in a rack was onlyaround 1 kW and cooling did not need to be a closely controlled activity, as simple whole roomcooling would suffice. But now with 1U servers and blade servers the potential heatgeneration is enormous. The average server has a running load of about 400 watts, meaningthat a 2 kW cooling capacity equates to only fiver servers per rack. Putting 42 of these serversin a rack, just because they fit, would develop over 16 kW of heat, and blade servers wouldgenerate over 20 kW.

Underfloor plenum cooling can supply about six kW of cooling capacity by the use of one ormore of the following upgrade methods;• Use a larger floor tile, up to 75% open area• Use a fan assisted floor grate• Use specialised blowers in the rack to bring more airflow into the rack and distribute it

across the front face of the equipment• Use rear doors on the racks that are full length blower units

Beyond about six kW, underfloor plenum cooling of racks becomes impractical and the nextstage is water-cooling of the entire rack.

Water is much more effective at removing heat than air. A water-cooled rack can dissipate inexcess of 20 kW of heat. These racks need to be plumbed into an existing chilled watergeneration and distribution system that would need to be placed outside of the equipmentroom. Liquid carbon dioxide cooling plants are also available now. CO2 is even more efficientthan water and can remove in excess of 30 kW of heat from a rack.

Directly cooled racks are thus much more efficient in terms of floor space used but they aremore expensive to buy, need plumbing in, and an external chiller plant still needs to be built.

For air conditioning applications for more than a medium sized rectangular computer room, itis advisable to use a computational fluid dynamics software program to model the airflow andcooling capacity of an HVAC design.


Data centre design

Airflow in a standard hot-aisle/cold aisle model

The diagram above shows the CRAC unit as the source of the chilled air and pumping it intothe underfloor plenum space. Air escapes into the cold aisle through the floor vents, passesthrough the racks, cooling them on the way, and appears in the hot aisle, where it rises. Itthen returns back to the CRAC unit to repeat the process. The CRAC units are located at theend of the hot aisles to facilitate the shortest return path back to the CRAC. Once the roomgoes over a certain size it is advisable to improve the return path by adding a ceiling plenum,with fans, to scavenge the hot air and direct it back to the CRAC units. It has been suggestedthat this would be beneficial once the floor area extends beyond 400 m2, although a dedicatedreturn plenum would benefit any size computer room.

Another item to take into account is locating the floor vents at the correct distance from theCRAC unit. Too close and the air velocity will cause a negative pressure at the vent relative tothe air in the room above and suck in hot air instead of blowing cold air out. The minimumdistance is about two metres before effective cooling takes place. The maximum distancefrom the CRAC unit again depends upon factors such as air volume from the CRAC unit, floordepth, obstructions, number and size of floor vents etc., but a figure of ten metres seems to becommonly accepted.

Some items, particularly communications equipment, are not designed for front-to-rear coolingbut side-to-side cooling, or even both at the same time!

Side-to-side items may be cooled by;• Placing in a low density environment on a two post frame with chilled air generally

supplied from a floor vent• Placing in a standard server rack with a front-to-side cooling converter fan fitted• Chilled water cooling matrices placed at the sides of the open frames that will allow

chilled air to be directed in a side-to-side direction


Data centre design

APC, a major supplier of IT air conditioning, offers the following estimating tool to helpcalculate the cooling capacity required of a computer room. Note the usual running loadshould be used for the IT equipment, not the nameplate rating, which is usually one thirdhigher than the normal running load.

The battery/UPS calculation is only required if the battery/UPS system is in the same computerroom. TIA 942 recommends that UPS systems greater than 100 kVA be placed in anotherroom.

Note that allowance should also be made for future expansion and redundancy in airconditioning calculations.

Fresh air

Even with air conditioning, the computer room needs to be ventilated. Air should be changedat least ten times per hour. British building regulations also require an air supply of ten litresper second per person, doubling if printers or photocopiers are in use.

Incoming air must be filtered with airborne particulate levels maintained within the limits ofFederal Standard 209E, Airborne Particulate Cleanliness Classes in Cleanrooms and CleanZones, Class 100,000.

Air from sources outside the building should be filtered using High Efficiency Particulate Air(HEPA) filtration rated at 99.97% efficiency (DOP Efficiency MIL-STD-282) or greater.

As the external temperature at British latitudes is below 22°C for about 70% of the year someof the huge electricity bills associated with cooling data centres can be mitigated by takingeven larger volumes of outside air during the autumn, winter and spring months, with theminimum ventilation rate maintained for the summer months.

Item Data required Heat output calculation Heat output subtotal

IT equipment Total IT load power Total IT running load, notin Watts nameplate values …………Watts

UPS with battery Power system rated (0.04 x power systempower in Watts rating) + (0.06 x total

IT load power) …………Watts

Power Distribution Power system rated (0.02 x power systempower in Watts rating) + (0.02 x total

IT load power) …………Watts

Lighting Floor area in sq m 21.5 W/sq m …………Watts

People Max No. of people 100 W per person …………Watts

Total …………Watts


Data centre design

10.0 Electrical systems

10.1 Mains input power requirementThe data centre must be supplied with sufficient electrical power to cope with ‘day-1’demands and foreseeable expansion plans. Suitable back-up equipment, e.g. UninterruptiblePower Supplies, UPS, and standby generators must also be considered.

The design goals are to specify;• The main electricity feed from the utility company• The distribution system around the data centre• Power distribution within the equipment racks• The UPS/Standby power system

The power distribution system also needs to be planned in accordance with the Tier 1 to 4requirements of TIA 942.

The first step is to understand the quantity of power required, at day one and when expansionplans are taken into account.

Some general rules;• ‘nameplate’ values can be derated by 33% for normal running power• UPS efficiency is typically 88%, i.e. 12% of the input power is consumed• Recharging UPS batteries need 20% of rated power• Lighting; allow 21.5 W per square metre• Air conditioning can take 100% of its rated cooling capacity

The normal I.T. running load is therefore the sum of all the nameplate ratings in all of theequipment, multiplied by about 0.67.

To size the power supply requirements however a number of conservative assumptions aremade such as allowing for the inrush current when the equipment starts and general overatingfactors as a margin of safety.


Data centre design

1. Add up all the nameplate ratings of all the equipment and multiply by 0.67, this is the day1 running load

2. Multiply this by whatever expansion factor is expected to apply to the data centre3. Add 50% to the above to allow for inrush current4. Add 32% to allow for the UPS inefficiency and battery charging requirement5. Add 21.5 W per square metre of floor space to allow for lighting6. Double the amount reached so far to allow for air conditioning power requirements7. Multiply the total so far by 1.25 to provide a further overating factor, so that cables aren’t

expected to work at their full safe load8. Add a figure; say 5% for power factor correction*. Modern I.T. equipment is usually

power factor corrected, but there will be some power factor loss

The figure thus arrived at is the amount of power that needs to be available in the data centre,even though it is unlikely to need this full amount under normal conditions. This figure alsoleads to correct choice of the standby generator.

Let’s take the example of a 200 square metre computer room with a day one nameplate loadof 100 kW and a required expansion capacity of 100%.

Day one running load = 100 x 0.67 = 67 kWLong term load, after expansion = 67 x 2 = 134 kWAdd 50% for peak load factor = 134 x 1.5 = 201 kWAdd 32% for UPS inefficiency and battery charging = 201 x 1.32 = 265 kWAdd 21.5 W/m2 for lighting = 265+(200x.021)=269 kWDouble this amount for power to run the air con = 269 x 2 = 538 kWMultiply by 1.25 for the overating factor = 538 x 1.25 = 672 kWAdd 5% for power factor correction = 672 x 1.05 = 706 kW

So we can see that the power supply to be designed in is more than ten times the day-onerunning load.

*Power factor. Remember that current times voltage equals volt-amperes, usually expressed as kVA.

Useful work, or power, is measured in watts, and volts x amps x power factor = watts. The power

factor is the cosine of the phase difference between the voltage and the current in an alternating

current circuit. This phase separation is caused by a reactive, i.e. capacitive or inductive, load. UPS

systems are always measured in kVA output, as they do not know the power factor of the load they will

be connected to, and hence the real power, in watts, deliverable.


Data centre design

10.2 UPS and backup requirementsHaving understood the sizing implications the next step is to consider the methods of back-upand redundancy and how this fits in with the Tiering philosophy of TIA 942.

TIA 942 summary

Remember that;‘N’ means only enough items to do the task at hand. Any one point of failure will stop the system‘N+1’ means one more item than is necessary, thus allowing for one point of failure‘2N’ means two complete independent paths

Going to 2N, or even better 2N+1, will give the required resilience that a data centre needs butobviously at some major cost, and not surprisingly 2N costs at least twice that for theprovision of the minimum required service.

An uninterruptible power supply system (UPS) needs to be defined to back up the powersupply system. This is usually based on batteries and a double conversion on-line UPS. Inthis method the incoming AC is rectified and permanently charges a battery pack which is alsoconnected in parallel back into an inverter, to make available the mains voltage AC again. Thisis a very reliable method and also isolates the I.T. load from sags, surges, spikes and mostharmonics coming in from the mains supply. The downside of this method is that it is veryinefficient with up to 12% of the input power wasted in the rectification-inversion cycle.

Other kinds of UPS are available and one is based on the kinetic energy of a large rotatingmass connected to a device which acts as a motor when input power is available and agenerator when the AC input fails. The kinetic energy stored in the rotating flywheel will thenproduce electricity for a short time. Kinetic energy devices are smaller and cheaper and haveless maintenance associated with them but usually have back-up times measured in tens ofseconds rather than the minutes offered by a battery system.

Tier 1 Tier 2 Tier 3 Tier 4

No. of delivery 1 1 1 active and 2 activepaths 1 passive

Utility entrance Single feed Single feed Dual feed Dual feed from different substations

Equipment Single cord with Dual cord with Dual cord with Dual cord withpower cords 100% capacity 100% capacity on 100% capacity 100% capacity on

each cord on each cord each cord

Generator fuel 8 hours, but no 24 hours 72 hours 96 hourscapacity generator required

if UPS backup timeis more than8 minutes

Redundancy N N+1 N+1 2N


Data centre design

UPS design options and requirements;1. Size the electrical power required, in kVA2. Decide what is the critical load that needs to be backed up with a UPS. Some people

include the air conditioning, and some don’t, expecting that the back-up generator will beonline before the equipment overheats. Backing up the aircon with the UPS will doublethe size of the UPS

3. Decide upon the length of time the battery pack needs to backup the system. Batterypacks are expensive, heavy and take a lot of space, recommendations are;

a. TIA 942, 5-30 minsb. SUN, 15 minutesc. Note that TIA 942 also specifies that a Tier 1 system does not need a

generator if the battery system can backup for at least 8 minutes4. Decide upon the level of redundancy desired/affordable, e.g. N, N+1, 2N or 2(N+1)5. Decide upon the location of the UPS and battery equipment. It should be close to the IT

equipment and main power feed to reduce cable losses. TIA 942 recommends that UPSsystems larger than 100 kVA should be located in their own separate room

6. Decide upon size and location of the standby generator. It must be in a secure position,and in an area where noise and fumes will not be disruptive. It should also be close tothe UPS system and switchgear to minimise cable losses

10.3 Electrical distribution around the computer roomThe electrical cabling, of adequate size to meet current and future design, must feed eachequipment rack location and planned location. For Tier 2 and above there must be duplicate,redundant feeds to each location.

Cabling may be fed into the top or bottom of racks, or both. Cabling run in the underfloorplenum space should be laid in the cold aisle at low level. Cabling entering through thebottom of the rack should be sealed with a brush strip to prevent entry of chilled air in anuncontrolled manner.

Cable should be terminated and presented on IEC 60309 connectors, of appropriate size forthe current and suitable for single or three phase connection as appropriate. Usual ratings are16 or 32 amp. The higher power ratings of today’s servers would suggest that two 32 ampfeeds would be required, giving around 7 kW. Higher power rating would require a three-phaseconnection, providing around 22 kW.

10.4 Electrical distribution within the rackAt its simplest the IEC 60309 connector is connected to a power strip which distributes theelectricity to a number of standard sockets, which in the UK would be a standard 13 amp BS1363 socket or an IEC 60320 socket. In America the plugs and sockets would be defined inthe NEMA series.


Data centre design

The power distribution units can extend beyond simple distribution of the power and may offer;• Sequential start up to lower inrush current• Simple filtering• Monitoring of current with an LED readout• Automatic switching between feeds• Network reporting and remote control ability through a TCP/IP connection

Other systems around take in mains voltage and distribute 48 V D.C around the rack to removethe need for each item of equipment to have its own dedicated power supply.

Earthing, bonding and the Signal Reference Grid

Earthing is required for three reasons;• Safety from electrical hazards• Reliable signal reference within the entire information technology installation• Satisfactory electromagnetic performance of the entire information technology installation

Correct earthing is required by law and described in various standards such as;• BS 6701 Telecommunication cabling and equipment installations• BS 7671 Requirements for electrical installations: IEE wiring regulations

16th Edition

Across Europe there is also;• EN 50310 Application of equipotential bonding and earthing in buildings

with information technology equipment• EN 50174-2 Information technology – Cabling installation – Part 2 –

installation and planning practices inside buildings

Across the world we have;• IEC 60364-1 Electrical installations of buildings, various sections including;

Part 5-548: Earthing arrangements and equipotential bondingfor information technology equipment

• ISO 11801:2002 Information technology – cabling for customer premises• ANSI/TIA/EIA-J-STD-607 Commercial building grounding and bonding requirements for

telecommunications

And from the world of telecommunications there is;• ETS 300 253 Equipment engineering – earthing and bonding of

telecommunications equipment in telecommunication centres• ITU-T K.27 Bonding configurations and earthing inside a

telecommunications building• ITU-T K.31 Bonding configurations and earthing of telecommunications

installations inside a subscriber’s building

And a particular standard referenced by TIA 942 is;• IEEE STD 1100-1999 Powering and Grounding Sensitive Electronic Equipment

11.0


Data centre design

It is essential that all metallic elements are correctly earthed according to the most relevantstandard above. This includes all equipment racks, cable containment and the metallicsheaths and armour of communications cables.

Note that whereas ‘earthing’ means, “the connection of the exposed conductive parts of aninstallation to the main earthing terminal of that installation (BS 7671)”, ‘bonding’ means, “theelectrical connection putting various exposed conductive parts and extraneous conductiveparts at a substantially equal potential (EN 50174-2).” Thus the connection for bonding mustbe capable of offering low enough impedance that a potential difference of not more then 1-volt rms can be maintained across the frequency range of interest.

This leads on to the requirement for the Signal Reference Grid, SRG, or a System ReferencePotential Plane, SRPP, as it is referred to in CENELEC standards.

The SRG is there to offer a suitable low-impedance path to ground for high frequencyinterference signals that cannot be achieved by simple ‘earthing’.

No standard mandates an SRG but everybody seems to recommend one, e.g.;• TIA 942. “Consideration should be given to installing a common bonding

network (CBN) such as a signal reference structure as described inIEEE Standard 1100 for the bonding of telecommunications andcomputer equipment”

• HP/Dell Site preparation guide. “If the system is on raised flooring, use a 2-footx 2-foot (61-cm x 61-cm) grounding grid”

• EN 50310 “A system reference potential plane (SRPP) conductive solid plane, asan ideal goal in potential equalising, is approached in practice byhorizontal or vertical meshes. The mesh width thereof is adapted tothe frequency range to be considered. Horizontal and vertical meshesmay be interconnected to form a grid structure approximating to aFaraday cage”

• SUN “A signal reference grid should be designed for the computer room.This provides an equal potential plane of reference over a broad bandof frequencies through the use of a network of low-impedanceconductors installed throughout the facility”

The SRG should therefore be constructed on the floor below the IT equipment and beconstructed of copper tapes approximately 50 mm wide. The dimensions of the grid havetypically been 24 x 24 inches (610 x 610 mm), however this only effectively gives protection upto around 30 MHz.

With gigabit Ethernet operating at up to 100 MHz this needs to be reduced to 200 mm to beeffective whereas ten gigabit Ethernet, operating at 500 MHz, would ideally need an almostcomplete surface. When using 50-mm copper tape a grid spacing of about 100 mm is thepractical limit.

The SRG must be effectively bonded to the building steel and the main electrical andtelecommunications grounding busbar, and all items on top or crossing the SRG must beconnected to it.


Data centre design

Fire detection, alarm and suppression

A fire design policy operates over a number of areas, all of which are related.• Design the building with materials and designs that minimise fire risk• Operate the building with practices that reduce fire risk• Detect fire and smoke with suitable apparatus• Sound an alarm if fire is detected to evacuate a building, summon the fire brigade and set

off fire extinguishants• Suppress the fire with automatic fire extinguishants

The principle fire safety legislation in the UK is the Fire Precautions (Workplace) Regulations1997/1999. This is obviously a major subject and one subject to laws and building regulations.TIA 942 Telecommunications Infrastructure Standard for Data Centers, April 2005, requires thefollowing for a data centre;

12.1 DetectionThe recommended smoke detection system for critical data centers where high airflow ispresent is one that will provide early warning via continuous air sampling and particle countingand have a range up to that of conventional smoke detectors.

……the system has four levels of alarm that range from detecting smoke in the invisible rangeup to that detected by conventional detectors. The system at its highest alarm level would bethe means to activate the pre-action system valve.

One system would be at the ceiling level of the computer room, entrance facilities, electricalrooms, and mechanical rooms as well as at the intake to the computer room air-handling units.

A second system would cover the area under the access floor in the computer room, entrancefacilities, electrical rooms, and mechanical rooms.

A third system is also recommended for the operations center and printer room to provide aconsistent level of detection for these areas.

A fire alarm system consists of

1. Detectorsa) Smoke, heat, flame etc

2. Manual call points3. Alarms

a) Bells, sirens, voice recording, visual etc4. Approved fire survival cable to link it all together5. A central control box to link it all together and to connect to other services

12.0


Data centre design

In the UK fire detection is governed by;BS 5839-1:2002 Fire detection and fire alarm systems for buildings. Code of practice for

system design, installation, commissioning and maintenance

And specifically for computer rooms and other electronic installations;BS 6266:2002 Code of practice for fire protection for electronic equipment installations

Fire alarm and detection components are generally covered by;BS EN 54 Fire detection and fire alarm systems

Typical fire detection and alarm loop

The cables must be fire survivable as described in BS5839-1: 2002 Clause (26.2 d & e) whichinvokes, amongst others;

• BS 60702-1: 2002 Mineral insulated cables and their terminations with a rated voltagenot exceeding 750V

• BS 6387:1994 Performance requirements for cables required to maintain circuitintegrity under fire conditions

Fire detectors come in a number of guises such as ionising smoke detectors, optical detectors,flame and heat detectors etc, but the smoke detection system recommended for computerrooms is a highly sensitive system that gives very early warning and is known as AspiratingSmoke Detection, ASD.

BS 6266-2002 recommends, “A dedicated smoke detection system interfaced with the mainbuilding system, …and an aspirating smoke detection to monitor return air flows,” for criticalequipment areas such as “centralised computer facilities.”

BS 5839 describes many different types of smoke and flame detectors and most importantly,where they should be sited. The siting of aspirating smoke detector inlets follows exactly thesame rules as more conventional smoke detectors.


Data centre design

ASD is a high sensitivity, aspirating type laser-based optical smoke detection system thatcontinually draws air within the protected area through a network of pipes where it is passedthrough a calibrated detection chamber. It is capable of providing very early warning of fireconditions thereby providing invaluable time to investigate and respond to a potential threat offire. ASD is very often referred to by a brand name, VESDA, Very Early Smoke DetectionApparatus. VESDA is a trademark of Vision Products Pty Ltd of Australia.

A ‘VESDA’ system can detect a fire within 70 seconds and activate a fire suppression responsein under two minutes. A sprinkler system would take four to six minutes under the samecircumstances.

ConclusionVarious standards, such as TIA 942 and BS 6266 recommend aspirating smoke detectors fordata processing applications such as data centres because of their quick reaction time.

The detection system should be able to give various levels of alarm and needs to be optimisedfor the different areas encountered within a data centre.

A data centre should have two levels of fire detection and suppression. An aspirating smokedetector linked to a gaseous fire suppression system as the first response and a pre-actionsprinkler system as the last resort.

12.2 Fire suppressionAccording to the SUN Data Centre Guide the ideal system would incorporate both a gassystem and a pre-action water sprinkler system in the ambient space.

According to the Fire Safety Advice Centre, (http://www.firesafe.org.uk/advicent.htm) thefollowing methods are considered for computer rooms;

Telecom Computer Control Rooms

Automatic sprinklers yes

Detection and pre-action sprinkler yes

Detection and water sprays (mist) no

Detection and total flood CO 2 yes

Foam Yes (under floor)

High sensitivity smoke detection aspirating systems yes

Detection and dry powder no

Detection and manual intervention yes

Detection and inert gas yes

Detection and fine particulate aerosol no

Detection and halocarbon gas yes


Data centre design

12.3 Gas suppressionEC Regulation 2037/2000 prohibits the sale and use of halons, including material that has beenrecovered or recycled, from 31st December 2002. Furthermore, with the exception ofequipment deemed critical under the Regulation, all fire-fighting equipment in the EUcontaining halons must be decommissioned before 31st December 2003.

The halon replacement market for clean agent gaseous suppression systems splits into inertgasses and halocarbon gasses.

Inert Gases Inert gas agents are electrically non-conductive clean fire suppressants that are used in designconcentrations of 35-50% by volume to reduce ambient oxygen concentration to between 14and 10%. Oxygen concentrations below 14% will not support the combustion of most fuels(and human exposure must be limited).

Halocarbon Gas SystemsA number of fire extinguishing halocarbon gases with zero ozone depletion potential (ODP)have been developed. These include both HFCs (hydrofluorocarbons) and PFCs(perfluorocarbons).

The DETR has published a document to give guidance on halon replacements, Advice onAlternatives and Guidelines for Users of Fire Fighting and Explosion Protection Systems.Although products are not officially approved or recognised by this route.

Inert gasses

Halocarbon gasses

In general, inert gas systems appear to take up more space and be slightly more expensivethan the halocarbon alternatives.

Trade Name Designation Gas Blend

NN100 IG-100 Nitrogen

Argotec IG-01 Argon

Argonite IG-55 Nitrogen/Argon mixture

Inergen IG-541 Nitrogen/Argon/Carbon dioxide mixture

Trade Name Designation Chemical Formula Chemical Name

FE-13 HFC 23 CHF 3 Trifluoromethane

FE-125 HFC 125 CF3 CHF2 Pentafluoroethane

FM-200 HFC 227ea CF3 CHFCF3 Heptafluoropropane

FE-36 HFC 236fa CF3 CH2 CF3 Hexafluoropropane

CEA-308 PFC-2-1-8 C3 F8 Perfluoropropane

CEA-410 PFC-3-1-10 C4 F10 Perfluorobutane


Data centre design

Manual means of fire suppression system discharge should also be installed. These shouldtake the form of manual pull stations at strategic points in the room. In areas where gassuppression systems are used, there is normally also a means of manual abort for thesuppression system.

See also;BS 6266:2002 Code of practice for fire protection for electronic equipment installationsBS ISO 14520-1:2000 Gaseous fire-extinguishing systems. Physical properties and system

design. General requirements

12.4 Pre-action sprinkler systemsThe gaseous fire suppression is seen as the first line of defence. After that comes the sprinklersystem. This must be of the pre-action type. This means that the pipes are normally dry, andcannot therefore drip onto the equipment.

The smoke detection system can set the first phase of the sprinkler system by letting waterenter into the piping but it still need the additional heat of the fire to set off the sprinklersthemselves. This is sometimes known as a ‘double-knock’ system.

12.5 Portable fire extinguishersPortable fire extinguishers should also be placed strategically throughout the room. Theseshould be unobstructed, and should be clearly marked. Labels should be visible above the tallcomputer equipment from across the room. Appropriate tile lifters should be located at eachextinguisher station to allow access to the subfloor void for inspection, or to address a fire. Atorch should also be located with the tile lifter.

ConclusionThe fire safety plan is a multilayered approach that requires a coordinated plan for• Designing for low flammability and fire risk• Operating with low risk• Emergency exits• Emergency lighting• Emergency exit signage• Fire detection, appropriate to the area covered• Fire alarm• Multi-level automatic fire suppression• Manual fire alarm and portable fire extinguishers• Staff training and fire drills in place• Maintenance plan for all equipment involved


Data centre design

13.0 Communications cabling and containment

Cabling is required to connect all the coomunications and control devices within the datacentre and the world beyond. Correct choice and installation of the cabling is essential toguarantee error-free transmission of data.

The communications protocols within the data centre nowadys revolve mostly around Ethernetand Fibre Channel. Communications speeds of at least 1 Gb/s should be designed for andnow ten gigabit speeds need to be considered. Design issues revolve around the selection ofcopper and/or optical fibre, grades of copper and fibre to be used, screened or unscreenedcopper cabling and levels of redundancy and resilience to be built in to the cabling model.

13.1 Spaces and hierarchyThe TIA 942 model shows the ‘Spaces’ that need to be accommodated and the cablinginterconnection hierarchy between and within them.

EN 50173-5 (Draft) is very similar but uses slightly different terminology.

TIA-942 EN 50173-5

Cross connect in the entrance room ENI (external network interface)

Main cross-connect in the MDA (main distribution area) MD (main distributor)

Horizontal cross-connect in the MDA or HDA (horizontal distribution area) ZD (zone distributor)

Zone outlet or consolidation point in the ZDA (zone distribution area) LDP (local distribution point)

Outlet in the EDA (equipment distribution area) EO (equipment outlet)

Horizontal cabling Zone distribution cabling

Backbone cabling (between MDA and HDAs) Main distribution cabling

Backbone cabling (from MDA to entrance room or from MDA to telecom room) Network access cabling

Telecommunications room Distributor


Data centre design

Alignment of terminology

13.2 Cable selectionTIA 942 recognises• 100-ohm twisted-pair cable (ANSI/TIA/EIA-568-B.2), Category 6 recommended UTP or

ScTP (ANSI/TIA/EIA-568-B.2-1);• Multimode optical fibre cable, either 62.5/125 micron or 50/125 micron (ANSI/TIA/EIA-

568-B.3), 50/125 micron 850 nm laser optimized multimode fibre is recommended(ANSI/TIA-568- B.3-1);

• Single-mode optical fibre cable (ANSI/TIA/EIA-568-B.3).• Coaxial media are 75-ohm (734 and 735 type) Telcordia Technologies GR-139-CORE) and

coaxial connector ANSI T1.404

EN 50173-5 recognises any of the cabling media addressed in EN 50173, e.g. Cat 5, Cat 6,Cat 7 etc, but Class E/Cat 6 is recommended for the main distribution and zone distributioncabling.

It would seem that within the Data Centre/Computer Room, a cable less than Category 6performance should not be used. Note that the American standards do not recogniseCategory 7/Class F.

None of the standards discuss 10GBASE-T or the forthcoming Augmented Category 6standard as this has not yet been published, or even finalised at the time of writing, but isexpected later in 2006.

Products claiming Cat6A performance are already on sale but whether unscreened (UTP)products can meet the alien crosstalk requirements and EMC regulations when operating atthe 500 MHz frequencies invoked by 10GBASE-T is still a matter of debate within the industry.Certainly a screened Cat 6 or Cat6A system is going to cope much better with the EMC andAlien crosstalk issues.


Data centre design

Cable selection issues• Copper cable

o At least Category 6. Consider Cat6A or Cat 7 for higher bandwidth performance.o Consider unscreened or screened. Unscreened is cheapest and seems to cope with

gigabit Ethernet speeds. Consider screened for severe EMC problems or upgrade to10GBASE-T operation

o Consider the fire performance of the cable. Unlike the USA there are no rulesrequiring very low flammability cabling in Europe. As a minimum request zerohalogen/low flammability cable to;• IEC 60332-3C

o The best performing cable in a fire situation is the plenum style meeting• NFPA 262: Standard Method of Test for Flame Travel and Smoke of Wires and

Cables for use in Air-Handling Spaces:2002• Or its higher performing companion known as Limited Combustible Plenum

cable

• Optical fibreo ISO 11801 and EN 50173 now classify optical fibres as OM1, OM2, OM3 and OS1.

OM means multimode fibre and OS means singlemode fibreo OM3 is a very high bandwidth fibre optimised for ten gigabit operation and is the

obvious choice for new data centre installationso Singlemode fibre, OS1, is not needed within the data centre but it may be needed to

connect to the outside world of telecommunications and should be put in place toallow for direct high speed communications from routers and SAN devices

o Optical connectors must also be specified. There are many to choose from and areStandards recognised. The market leader for high speed data communications isnow the LC connector

13.3 Preconnectorised cablingCabling is traditionally installed, as cable, which is then terminated on-site in patch panels,outlets and other connectors. There is a big time advantage to be gained by terminating thecables off-site and installing the ready-made assemblies into the data centre.

Preconnectorised cabling is most popular when time on site is at an absolute premium. Thismay be in a new build, such as a data centre, where time scales are critical and many differenttrades are vying for the right to work on any particular bit of floor space at any time.

Other time-critical areas are live sites that need additional cabling but where the costs andimplications of downtime are horrendous, such as a trading floor or call centre. Such a facilitymay want to have all its cabling upgraded or extended in one overnight operation.

Busy city centre facilities will also suffer from a lack of parking and loading bays, on-sitestorage restrictions and security worries associated with cable installers needing weeks ofaccess time to the site.

Preconnectorised cabling should reduce time needed on site by around 75% compared totraditional installation.


Data centre design

Quality of the terminations should also be improved by allowing sophisticated Category 6copper and optical fibre terminations to be made in a clean factory environment by skilledpeople. Each cable assembly can be 100% checked in the factory and whatever is sent tosite is known to be of the highest quality.

There are no particular disadvantages to preconnectorised cabling, and it should be cost-neutral to the end–user, however accurate surveys need to be carried out to ensure correctcable lengths are made up and installed.

Connectix Express preconnectorised copper cabling

13.4 Cable containmentThe cable containment must protect the cables and also the bend radius requirements of thecables. Containment may take the form of basket, trays, conduit, trunking etc. If it is metallic,then all of the containment must be correctly earthed.

All cabling, patch panels, earthing and containment system must be adequately labelled andmarked and records kept. This aspect of cabling is described in the following;

ANSI/TIA/EIA-606-A Administration Standard for the Telecommunications Infrastructure ofCommercial BuildingsEN 50174-1 Information technology – cabling installation – Part 1:Specification and qualityassuranceISO/IEC 14763-1: Information Technology – Implementation and operation of customerpremises cabling – Part 1:AdministrationTIA 942 Telecommunications Infrastructure Standard for Data Centersand

5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

13 14 15 16 17 18 19 20 21 22 23 241 2 3 4 5 6 7 8 9 10 11 12

1 2 3 4

Cable APanel 01

Cable BPanel 01

Cable CPanel 01

A010

1

B010

5

C01

09

A010

2

B010

6

C01

10

A010

3

B010

7

C01

11

A010

4

B010

8

C01

12

Cable APanel 02

A0201

A0202

A0203

A0204

Ca

bel

CD

esk

01C

ab

elB

Fol

or0

1 BF014

BF013

BF012

BF011

CD014

CD013

CD012

CD011

Panel topanel link Panel to

desk link

Panel tofloor link

Desk Pod

Floor Box


Data centre design

BS 6701:2004 Telecommunications equipment and telecommunications cabling —Specification for installation, operation and maintenance

… that all cables and components be suitably marked to uniquely identify them. Thedurability of all labelling must also be suitable for the rigours of the environment in whichthey are placed and the expected timescale of the installation, usually in excess of tenyears.

The cables need to be contained and protected and separated from other services. Forexample EN 50174-2 requires a separation of at least 200 mm between unscreened data andunscreened power cables, although distances can come down if any of the cables arescreened. BS6701 requires a 50 mm separation at all times between cables unless there is anon-metallic divider separating the two groups. In the UK, BS6701 and EN 50174-2requirements need to be overlaid and the worst-case separation distances used for a correctinstallation.

BS6701 and EN 50174-2 overlaid

13.5 Cabling standards – summaryAt present EN 50173 defines the cabling design. Soon the more specific EN 50173-5 standardwill more precisely define data centre cabling requirements. On a wider basis, ISO11801 andANSI/TIA/EIA-568-B also define cable system design.

TIA 942 defines the cabling hierarchy for data centres and states the permissible range ofcables. TIA 942 only invokes other American standards such as ANSI/TIA/EIA-568-B.

EN 50174 parts 1,2 and 3 describe installation and quality assurance techniques.

EN 50310 describes the equipotential bonding system for information technology installations.

EN 50346 describes the testing methodology to prove compliance of the installed cabling.

Type of Installation Separation Distance

Without a With a non- Aluminium Steeldivider metallic divider divider divider

Unscreened power cable 200 mm 200 mm 100 mm 50 mmand unscreened IT cable

Unscreened power cable 50 mm 50 mm 50 mm 50 mmand screened IT cable

Screened power cable 50 mm 30 mm 50 mm 50 mmand unscreened IT cable

Screened power cable 50 mm 0 mm 50 mm 50 mmand screened IT cable


Data centre design

13.6 Modular DesignsData Centre users rarely know exactly what the format of the I.T. equipment will be when theData Centre goes live and certainly don’t know what we will be expected of it next year. Forthis reason many people like to design a generic centre based on flexible modular units like theCapitoline Cluster Concept (www.capitoline.co.uk). In the example shown below a clusterconsists of five server racks with half of one rack dedicated to cabling interconnection. Eachrack takes 60 Cat 6 cables and one OM3 8-fibre cable back to the Main Distribution Frame.One cluster is dedicated to Wide Area Networking/Router/Telecoms applications. It too has 60Cat 6 cables but more optical fibre and also a singlemode link back to the MDF to allow fordirect high-speed connection into the outside world. The Storage Area Network, SAN, clusteris identically cabled. The MDF mirrors the server, WAN and SAN zones and also has adedicated area to connect to the Telecoms Room and ENI. For additional resilience eachServer cluster has Cat 6 cables wired directly to the WAN and SAN clusters.

Modular designs and cluster concepts are bound to be more popular as the rate of change inData Centres increases. The cluster concept incorporates the air conditioning as well by ratingeach rack with a minimum 2 kW load dissipation and a planned upgrade path up to 20 kW perrack.


Data centre design

14.0 Security, Access Control and CCTV

TIA 942 requires that the Data Centre be secure.

CCTV requirements

Security Access Tier 1 Tier 2 Tier 3 Tier 4Control/monitoring at:

Generators Industrial grade Intrusion Intrusion Intrusionlock detection detection detection

UPS. Telephone Industrial grade Intrusion Card access Card access& MEP rooms lock detection

Fibre vaults Industrial grade Intrusion Intrusion Card accesslock detection detection

Emergency Exit Doors Industrial grade monitor Delay egress Delay egresslock

Accessible exterior Off site Intrusion Intrusion Intrusionwindows monitoring detection detection detection

Security operations centre N/a N/a Card access Card access

Doors into computer Industrial grade Intrusion Card or Card or rooms lock detection biometric access biometric access

Perimeter building doors Off site Intrusion Card access Card accessmonitoring detection

Doors from lobby Industrial grade Card access Single person Single personto floors lock interlock interlock

CCTV monitoring Tier 1 Tier 2 Tier 3 Tier 4

Building perimeter No requirement No requirement Yes Yesand parking

Generators N/a N/a Yes Yes

Access controlled doors No requirement yes Yes Yes

Computer room floors No requirement No requirement Yes Yes

UPS, telephone and No requirement No requirement Yes YesMEP rooms

CCTV Tier 1 Tier 2 Tier 3 Tier 4

CCTV recording on No requirement No requirement Yes; digital Yes; digitalall cameras

Recording rate, N/a N/a 20 f/s min 20 f/s minframes per second


Data centre design

15.0 Building Management Systems

Building Management Systems, or BMS, can cover a range of technologies that controls andoptimises space heating, air conditioning, hot water service and lighting in buildings.

TIA 942 makes the following statement;A Building Management System (BMS) should monitor all mechanical, electrical, and otherfacilities equipment and systems. The system should be capable of local and remotemonitoring and operation.

Individual systems should remain in operation upon failure of the central BuildingManagement System (BMS) or head end. Consideration should be given to systemscapable of controlling (not just monitoring) building systems as well as historical trending.24-hour monitoring of the Building Management System (BMS) should be provided byfacilities personnel, security personnel, paging systems, or a combination of these.Emergency plans should be developed to enable quick response to alarm conditions.

We can consider a Data Centre as being in three layers for the BMS requirement;• Incorporation into a larger and pre-existing site BMS• A BMS dedicated to the Data Centre facility• Rack level monitoring and control

With IP based networks more and more of these systems come together with one commoncabling system. The exception is the fire detection loop cabling which must be dedicated andfire survival grade. Many of the control systems rely on automation protocols such asLONWorks and BACNET to communicate and control the end equipment but the higher levelsof communication between controllers is now reliant upon TCP/IP and Ethernet.

The environmental monitoring parameters are;• Temperature • Access• Smoke • Vibration• Water • Air flow• Humidity • Particles in the incoming air flow

CCTVAccess Control & MonitoringFire alarmsBMS -HVAC

-LightingEnvironmental monitoring

Building basedRoom basedRack based

Common IP cablingDedicated cablingLocal alarm/controlRemote alarm/control


Data centre design

16.0 Project Management and other issues

So far in this document we have considered the various ‘Tiering’ levels defined in TIA 942 andfrom the Uptime research institute. A data centre does not need to be on the same Tier forevery facility. It is quite acceptable for the installation to be Tier 2 for air conditioning and Tier 4for power supply for example. It all depends upon what the customer wants and can afford.

We can take further definitions from TIA 942

N - Base requirementSystem meets base requirements and has no redundancy

N+1 redundancyN+1 redundancy provides one additional unit, module, path, or system in addition to theminimum required to satisfy the base requirement. The failure or maintenance of any singleunit, module, or path will not disrupt operations

2N redundancy2N redundancy provides two complete units, modules, paths, or systems for every onerequired for a base system. ”Failure or maintenance of one entire unit, module, path, or systemwill not disrupt operations

2(N+1) redundancy2(N+1) redundancy provides two complete (N+1) units, modules, paths, or systems. Even inthe event of failure or maintenance of one unit, module, path, or system, some redundancy willbe not be disrupted

Tier I Data Center: BasicA Tier I data centre is susceptible to disruptions from both planned and unplanned activity. Ithas computer power distribution and cooling, but it may or may not have a raised floor, a UPS,or an engine generator. If it does have UPS or generators, they are single-module systems andhave many single points of failure. The infrastructure should be completely shut down on anannual basis to perform preventive maintenance and repair work. Urgent situations may requiremore frequent shutdowns. Operation errors or spontaneous failures of site infrastructurecomponents will cause a data center disruption.

Tier 1 Tier 2 Tier 3 Tier 4

Site availability 99.671% 99.749% 99.982% 99.995%

Downtime (hours/yr) 28.8 22.0 1.6 0.4

Operations Center Not required Not required Required Required

Redundancy for power, N N+1 N+1 2(N+1)cooling

Gaseous fire suppression Not required Not required Approved Approvedsystem system system

Redundant backbone Not required Not required Required Requiredpathways


Data centre design

Tier II Data Centre: Redundant ComponentsTier II facilities with redundant components are slightly less susceptible to disruptions fromboth planned and unplanned activity than a basic data centre. They have a raised floor, UPS,and engine generators, but their capacity design is “Need plus One” (N+1), which has a singlethreaded distribution path throughout. Maintenance of the critical power path and other partsof the site infrastructure will require a processing shutdown.

Tier III Data Centre: Concurrently MaintainableTier III level capability allows for any planned site infrastructure activity without disrupting thecomputer hardware operation in any way. Planned activities include preventive andprogrammable maintenance, repair and replacement of components, addition or removal ofcapacity components, testing of components and systems, and more. For large sites usingchilled water, this means two independent sets of pipes. Sufficient capacity and distributionmust be available to simultaneously carry the load on one path while performing maintenanceor testing on the other path. Unplanned activities such as errors in operation or spontaneousfailures of facility infrastructure components will still cause a data centre disruption. Tier IIIsites are often designed to be upgraded to Tier IV when the client’s business case justifies thecost of additional protection.

Tier IV Data Centre: Fault TolerantTier IV provides site infrastructure capacity and capability to permit any planned activitywithout disruption to the critical load. Fault-tolerant functionality also provides the ability of thesite infrastructure to sustain at least one worst-case unplanned failure or event with no criticalload impact. This requires simultaneously active distribution paths, typically in aSystem+System configuration. Electrically, this means two separate UPS systems in whicheach system has N+1Tier IV Data Centre: Fault Tolerant redundancy. Because of fire andelectrical safety codes, there will still be downtime exposure due to fire alarms or peopleinitiating an Emergency Power Off (EPO). Tier IV requires all computer hardware to have dualpower inputs as defined by the Institute’s Fault-Tolerant Power Compliance Specification.

Safety AuditThe installation must be audited for safety both at design stage, project handover and routineinspection. The requirements of the fire safety programme are already outlined in section 12.Additional safety audit points are;• Raised Floors (especially lifting tiles or tripping or falling)• Lifting Hazards• Electrical Shock Hazards• Static Discharge Hazards• Cutting Hazards• Pinching / Amputation Hazards• Fire Hazards• Accidental Triggering a gaseous Fire Retardant System Dump• Accidental Unplugging Network Cables or Power From Servers• Infra red laser hazard• Excessive noise


Data centre design

For the last point it is worth noting that sound levels at work in Europe were reduced inFebruary 2006. The EC Noise at Work Directive 2003/10/EC was made on 6th February 2003and repeals and replaces 86/188/EC as from (mainly) 15th February 2006.

Where is the money likely to go in a Data centre?

An American example

Example small data centre, courtesy Future-Tech, www.future-tech.co.uk


Data centre design

A Appendix I

Some Standards referenced in this documentANSI/TIA/EIA-568-B Commercial Building Telecommunications Cabling StandardANSI/TIA/EIA-606-A Administration Standard for the Telecommunications Infrastructure ofCommercial BuildingsANSI/TIA/EIA-J-STD-607 Commercial building grounding and bonding requirements fortelecommunicationsASHRAE Thermal Guidelines for Data Processing EnvironmentsBS EN 54 Fire detection and fire alarm systemsBS 5499-4:2000 Safety signs, including fire safety signs. Code of practice for escape routesigningBS 5266-1, The Code of Practice For Emergency lightingBS 5839-1:2002 Fire detection and fire alarm systems for buildings. Code of practice forsystem design, installation, commissioning and maintenanceBS 60702-1: 2002 - Mineral insulated cables and their terminations with a rated voltage notexceeding 750VBS 6387:1994 - Performance requirements for cables required to maintain circuit integrityunder fire conditionsBS 6266:2002 Code of practice for fire protection for electronic equipment installationsBS 6701 Telecommunication cabling and equipment installationsBS 7671 Requirements for electrical installations: IEE wiring regulations 16th EditionBS ISO 14520 P1: 2000(E), Gaseous fire-extinguishing systems. Physical properties andsystem design. General requirements,BS 8300:2001 Design of buildings and their approaches to meet the needs of disabled people— Code of practice, andBuilding Regulations 2000 Part M Access and facilities for disabled peopleDETR Advice on Alternatives and Guidelines for Users of Fire Fighting and ExplosionProtection SystemsEN 50310 Application of equipotential bonding and earthing in buildings with informationtechnology equipmentEN 50173 Information technology - Generic cabling systems -- Part 1: Generalrequirements and office areasEN 50174-1 Information technology – cabling installation – Part 1:Specification and qualityassuranceEN 50174-2 Information technology – Cabling installation – Part 2 – installation and planningpractices inside buildingsEN 50346 Information technology - Cabling installation - Testing of installed cablingEN 12825 Raised access floorsETS 300 253 Equipment engineering – earthing and bonding of telecommunicationsequipment in telecommunication centresFederal Standard 209E, Airborne Particulate Cleanliness Classes in Cleanrooms and CleanZones, Class 100,000IEC 60309 Plugs, socket-outlets and couplers for industrial purposes - Part 1: GeneralrequirementsIEC 60320 Appliance couplers for household and similar general purposes - Part 1: Generalrequirements


Data centre design

IEC 60332-3C Tests on electric cables under fire conditions - Part 3-10: Test for vertical flamespread of vertically-mounted bunched wires or cablesIEC 60364-1 Electrical installations of buildings, various sections including; Part 5-548:Earthing arrangements and equipotential bonding for information technology equipmentIEEE STD 1100-1999 Powering and Grounding Sensitive Electronic EquipmentNFPA 262: Standard Method of Test for Flame Travel and Smoke of Wires and Cables for use inAir-Handling Spaces:2002ISO/IEC 14763-1: Information Technology – Implementation and operation of customerpremises cabling – Part 1:AdministrationISO 11801:2002 Information technology – cabling for customer premisesITU-T K.27 Bonding configurations and earthing inside a telecommunications buildingITU-T K.31 Bonding configurations and earthing of telecommunications installations inside asubscriber’s buildingThe Property Services Agency (PSA) Method of Building Performance Specification 'PlatformFloors (Raised Access Floors)', MOB PF2 PSTIA 942 Telecommunications Infrastructure Standard for Data Centers, April 2005.VDI 2054, Air conditioning systems for computer areas

Documents

Data centre design · 2017-03-14 · room and its support areas,” according to TIA 942. This design guide is based upon the requirements of TIA 942 Telecommunications Infrastructure