Datacenter Planning Guide

8/8/2019 Datacenter Planning Guide

1/106

Sun Microsystems, Inc.4150 Network CircleSanta Clara, CA 95054 U.S.A.650-960-1300

Send comments about this document to: [email protected]

Sun Microsystems Data Center SitePlanning Guide

Data Centers Best Practices

Part No. 805-5863-13January 2003, Revision A


2/106

PleaseRecycle

Copyr ight 2003 Sun Microsystems, Inc., 4150 Netw ork Circle,Santa Clara, California 95054,U.S.A. All rights reserved .

Sun Microsystems, Inc. has intellectua l prop erty rights relating to technology embod ied in the prod uct that is described in this docu men t. Inparticular, and without limitation, these intellectual property rights ma y includ e one or more of the U.S. patents listed athttp:/ / ww w.sun.com/ p atents and one or more additional patents or pending patent ap plications in the U.S. and in other countries.

This documen t and the p roduct to which it pertains are distributed und er licenses restricting their use,copying, distribution, anddecompilation.N o part of the product or ofthis document may be reproduced in any form by any means without prior written authorization ofSun and its licensors, if any.

Third-pa rty softwa re, includin g font technology, is copyright ed and licensed from Sun sup pliers.

Parts of the prod uct may be der ived from Berkeley BSD systems, licensed from the University ofCa lifornia. UNIX is a registered tradem ark inthe U.S. and in other countries, exclusively licensed through X/ Open Comp any, Ltd.

Sun, Sun Microsystems, the Sun logo, AnswerBook2,d ocs.sun.com, and Solaris are trademar ks or registered trademar ks of Sun Microsystems,Inc. in the U.S. and in other coun tries.

All SPARC trad emar ks are used u nd er license and are trad ema rks or registered trad emar ks ofSPARC Interna tional, Inc. in the U.S.an d in othercountr ies. Prod ucts bearing SPARC trad emar ks are based upon an architecture develop ed by Sun Microsystems, Inc.

The OPEN LOOK and Sun Grap hical User Interface was dev eloped by Sun Microsystems, Inc. for its users and licensees. Sun acknow ledgesthe pioneering efforts of Xerox in researching and d eveloping the concept ofv isual or graph ical user interfaces for the compu ter industr y. Sunholds a non-exclusive license from Xerox to the Xerox Graph ical User Interface, wh ich license also covers Suns licensees who implement OPENLOOK GUIs and oth erw ise comp ly with Suns wr itten license agreements.

U.S. Government RightsCommercial use. Government u sers are subject to the Sun Microsystems, Inc.stand ard license agreement andapp licable provisions of the FAR and its sup plements.

DOCUMENTATION IS PROVIDED "AS IS" AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS AND WARRANTIES,INCLUDING ANY IMPLIED WARRANTY OF MERCHAN TABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NO N-INFRINGEMENT,ARE DISCLAIMED, EXCEPT TO TH E EXTENT TH AT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID.

Copyr ight 2003 Sun Microsystems, Inc., 4150 Netw ork Circle,Santa Clara, California 95054,Etats-Unis. Tous droits rservs.Sun Microsystems, Inc. a les droits de prop rit intellectuels relatants la techno logie incorpor e dans le prod uit qui est dcrit dan s cedocum ent. En particulier,et sans la limitation, ces droits de proprit intellectuels peuven t inclure un ou p lus des brevets amricains numrs http:/ / w ww.sun .com/ p atents et un ou les brevets plus sup plm entaires ou les app lications de brevet en attente dans les Etats-Unis et dansles autr es pays.

Ce produ it ou documen t est protg par un copyright et distribu avec des licences qui en restreignent lutilisation,la copie, la distribution, et ladcompilation.Au cune partie de ce produ it ou docum ent ne peut tre reproduite sous aucune forme, parquelque m oyen que ce soit, sanslauto risation pr alable et crite de Sun et d e ses bailleur s de licence,sil y ena.

Le logiciel dten u par d es tiers, et qui comp rend la technologie relative aux polices de caractres, est protg par un copyr ight et licenci par desfournisseurs de Sun.

Des parties de ce prod uit pou rront tre dr ives des systmes Berkeley BSD licencis par lUniversit de Californie. UNIX est une mar que

dp ose aux Etats-Unis et dan s dautr es pays et licencie exclusivemen t par X/ Op en Comp any, Ltd.Sun, Sun Microsystems, le logo Sun, AnswerBook2,d ocs.sun.com, et Solaris sont des marqu es de fabrique ou des m arques dp oses de SunMicrosystems,Inc. aux Etats-Unis et dans d autres pays.

Toutes les marqu es SPARC sont utilises sous licence et sont des ma rqu es de fabrique ou d es marqu es dposes d e SPARC Internation al, Inc.aux Etats-Unis et dans dautres pays. Les prod uits protan t les mar ques SPARC sont bass sur une architecture dvelop pe par SunMicrosystems,Inc.

Linterface dutilisation graphiqu e OPEN LOOK et Sun a t dv eloppe par Sun Microsystems, Inc. pou r ses utilisateur s et licencis. Sunreconnat les efforts de pionn iers de Xerox pour la recherche et le dvelopp men t du concep t des interfaces dutilisation visuelle ou grap hiqu epou r lindu strie de linforma tique. Sun dtient u ne license non exclusive do Xerox sur linterface dutilisation graphiqu e Xerox, cette licencecouvra nt galement les licencies de Sun qui metten t en place linterface d utilisation graph ique OPEN LOOK et qui en outre se conformen taux licences crites de Sun.

LA DO CUMENTATION EST FOURNIE "EN LTAT" ET TOUTES AUTRES CON DITIONS, DECLARATIONS ET GARAN TIES EXPRESSESOU TACITES SONT FORMELLEMENT EXCLUES, DANS LA MESURE AUTORISEE PARLA LOI APPLICABLE, YCOMPRIS NO TAMMENTTOUTE GARAN TIE IMPLICITE RELATIVE A LA QUALITE MARCHAND E, A LAPTITUDE A UN E UTILISATION PARTICULIERE OU ALABSENCE DE CON TREFAON .


3/106

3

Contents

Preface 13

1. General Site Selection Criteria 11

1.1 Locating the Building 11

1.2 Locating the Computer Room 13

2. General Computer Room Criteria 21

2.1 Designing the Room 21

2.1.1 Computer Aided Design (CAD) Drawings 21

2.1.2 Design Flexibility and Planned Redu ndan cy 22

2.1.3 Expansion Considerations and Preparations 22

2.1.4 Room Layout and Planning 23

2.1.5 Computer Room Access 24

2.2 Designing the Floor 24

2.2.1 Floor Height 25

2.2.2 General Supp ort Grid Recommendations 25

2.2.3 General Tile Construction Recomm endations 25

2.2.4 Floor Maintenance 26

2.2.5 Cutouts and Other Tile Customizations 26

2.2.6 General Load Rating Recomm endations 26


4/106

4 Sun Microsystems Data Center Site Planning Guide January 2003

2.2.7 Fire Rating 27

2.2.8 Supp lemental Bracing 27

2.3 Building the Room 272.3.1 Building Preparation 28

2.3.2 Building Materials Selection 29

3. Cooling and Air Distribution 21

2.1 Temperature Recommendations 22

2.2 Relative Hu midity Recommendations 22

2.3 Temperature and Relative Hum idity Problems 23

2.3.1 Thermal Concerns 24

2.3.2 Electrostatic Discharge (ESD) 24

2.3.3 Corrosion 24

2.4 Temperature and Relative Hum idity Monitoring 25

2.5 Air Conditioner and Hu midifier Design, Maintenance, and Placement 26

2.5.1 Air Conditioner and Hu midifier Set-points 29

2.6 Mechanical Supp ort Systems 210

2.7 Air Distribution Tile Placement 210

2.8 Hardware Placement 213

2.9 Subfloor Pressure Differential Recomm endations 217

2.10 Sup ply Air Plenum Integrity Concerns 218

2.11 Vapor Barrier Design and Conditions 219

4. Environmental Contaminants 41

4.1 Recommended Air Quality Levels 41

4.2 Contaminant Properties and Sources 43

4.2.1 Operator Activity 44

4.2.2 Hardware Movement 44

4.2.3 Outside Air 44


5/106

Contents 5

4.2.4 Stored Items 45

4.2.5 Outside Influences 45

4.2.6 Cleaning Activity 454.3 Contaminant Effects 46

4.3.1 Physical Interference 46

4.3.2 Corrosive Failure 46

4.3.3 Shor ts 46

4.3.4 Thermal Failure 474.4 Room Condit ions 47

4.5 Exposure Points 411

4.6 Filtrat ion 412

4.7 Positive Pressurization and Ventilation 413

4.8 Cleaning Procedures and Equipment 4144.8.1 Daily Tasks 415

4.8.2 Weekly Tasks 415

4.8.3 Quarterly Tasks 416

4.8.4 Bi-Annu al Tasks 416

4.9 Activity and Processes 417

5. Safety and Security 51

5.1 Fire Prevention in a Computer Room 51

5.2 Physical Structure 52

5.3 Fire Detection and Supp ression 52

5.4 Water Detection and Leak Precautions 54

5.5 Personnel Safety 55

5.6 Operator Health Considerations 55

6. Facility Power Requirements 61

6.1 Power System Design 61


6/106


6.1.1 Multiple Utility Feeds 62

6.1.2 Uninterrup tible Power Sup ply (UPS) 62

6.1.3 Backup Power Generators 626.1.4 Maintenance Bypass 62

6.1.5 Installation and Placement 63

6.1.6 Single-and Three-Phase Power 67

6.1.7 Separately Derived Systems 67

6.2 Grounding and Bonding 676.2.1 Ground 68

6.2.2 Recomm ended Acceptable Values 69

6.2.3 Equipment Grounding Condu ctor Impedance 69

6.2.4 Grounding of Building Structural Steel 69

6.2.5 Special Forms of Earth Grounding Electrodes 6106.2.6 Bonding of Metal Sleeves 610

6.3 Signal Reference Grid 610

6.3.1 Recommen ded Practices for Signal Reference Grid (SRG) 611

6.4 Input Power Quali ty 614

6.4.1 Power Conditioning Technology 6156.4.2 Voltage Tolerance 617

6.4.3 Frequency Tolerance 617

6.4.4 Harmonic Content 617

6.4.5 Branch Circuits 617

6.4.6 Voltage Spikes 6186.4.7 Lightning Protection 618

6.4.8 Emergency Power Control 618

6.5 Wiring and Cabling 619

6.6 Electromagnetic Compatibility (EMC) 620

6.7 Electrostatic Discharge (ESD) 621


7/106

Contents 7

6.7.1 ESD Damage 621

6.7.2 ESD Control 621

6.8 Site Surveys and Site Power Analyses 623

7. Receiving, Transporting, and Staging Guidelines 71

7.1 Unloading, Moving, Unpacking, and Storing Guidelines 71

7.2 Equipment Staging Area Guidelines 72

A. Conversion Information A1

B. List of References B1

B.1 References B1


8/106



9/106

9

Figures

FIGURE 2-1 Typical Upward Verses Downward Computer Room Air Flow Patterns 27

FIGURE 2-2 Cooling Short Cycle Air Patterns 212

FIGURE 2-3 Congested Cabling 213

FIGURE 2-4 Preferred Hardware Placement 215FIGURE 2-5 Alternate Hardware Placement 216

FIGURE 2-6 Subfloor Penetration 219

FIGURE 4-1 Floor Surface Contaminants Air Plenum Conditions. 47

FIGURE 4-2 Subfloor Penetration 48

FIGURE 4-3 Dirty Unsealed Subfloor 49

FIGURE 4-4 Well-sealed Subfloor 410

FIGURE 6-1 Example of Poor Equipment Installation and Placement 64

FIGURE 6-2 Example of Better Equipment Installation and Placement 65

FIGURE 6-3 Example of Best Equipment Installation and Placement 66

FIGURE 6-4 Typical Installations With Power Distribution Unit and Signal Reference Grid 613

FIGURE 6-5 Disorganized Cabling 620


10/106



11/106

11

Tables

TABLE 2-1 Environmental Requirements 23

TABLE 4-1 Gas Limit Recommendations 43

TABLE 4-2 Filter Efficiency Comparison 413

TABLE 4-3Cleaning Schedule 415

TABLE 6-1 FIPS PUB 94 Tolerances Chart 614

TABLE 6-2 Power Conditioning Technology 616

TABLE 6-3 Electrostatic Voltage At Workstations 622

TABLE A-1 Conversion Factors A1


12/106



13/106

13

Preface

This document p rovides the necessary information for constructing a suitable datacenter environment for a Sun Microsystems server.

AcknowledgmentsThanks to Worldwide Environmental Services (WES) for their work on the Sun DataCenter Site Planning Guide.

How This Book Is OrganizedChapter 1 provides information on general site selection criteria.

Chapter 2 provides information on designing and building a computer room.

Chapter 3 provides information on cooling an d air distribution.

Chapter 4 provides information on environmental contaminants.

Chapter 5 provides information on safety and security

Chapter 6 provides information on facility power requirements.

Chapter 7 provides information on receiving, transporting, and staging guidelines.

Append ix A provides conversion information.

Append ix B provides a list of references.


14/106


Accessing Sun Documentation OnlineYou can view, print, or purchase a broad selection of Sun documentation, includinglocalized versions, at:

http://www.sun.com/documentation

Sun Welcomes You r CommentsSun is interested in imp roving its documen tation and w elcomes your comm ents andsuggestions. You can email your comments to Sun at:

[email protected]

Please include t he p art n um ber (805-5863-13) of your d ocum ent in t he su bject line ofyour email.
http://a2-chapter11.pdf/http://a2-chapter11.pdf/http://a2-chapter11.pdf/http://a2-chapter11.pdf/


15/106

1-1

CHAPTER 1

General Site Selection Criteria

It is easier to address facility design issues in the planning stages than to correctproblems after the fact. Often, it is impossible to implement extensive actions in anon-line compu ter room withou t imp acting u ptime. For this reason, it is extremelyimportan t that adequ ate attention is paid to issues such as the physical location ofthe bu ilding, the location of the d ata center relative to the other a reas of thebuilding, and all aspects of the sup port infrastructure. A little extra p lanning cansave a tremend ous am ount of time, money and aggravation over the lifetime of thefacility.

1.1 Locating the Building

Cost and numerous external factors influence the site selection for a buildingaccomm odating a new data center. Not least amon g these factors is the potentialenvironmental imp acts the building will have on the data center. Often, a siteplanner weighs nu merous conflicting criteria wh en selecting the m ost app ropriatelocation. Whenever possible, consider the following factors.

s Natural D isasters; avoid sites in areas susceptible to natural disasters. Floodplains, tornado or hu rricane hot-spots or seismically active areas are n ot optimalchoices. While precautions can be taken to accommod ate building in such areas,

there are ad ditiona l cost consider ations, and m ore likelihood of imp act on utilitiesor other infrastructure support.s Electromagnetic Interference; avoid choosing a site near sources of

electromagn etic interference (EMI) or Radio Frequ ency Inter ference (RFI).Telecommu nications signal facilities, airports, electrical railways, or oth er similarsites are often associated with high levels of EMI/ RFI that could interfere withthe prop er functioning of comp uter h ardw are. Shielding comp uter room s fromEMI/ RFI adds to the construction cost.


16/106

1-2 Sun Microsystems Data Center Site Planning Guide January 2003

s Industrial Pollution; avoid locating the facility near major sources of industrialpollution, such as factories, manufacturing facilities, or sewage treatmentfacilities. Chemicals associated with these facilities can impact hardwarereliability if they migrate to the controlled areas of the data center. Even thechemicals associated with field treatment in large agricultural areas have beenknow n to cause hard wa re failures. While these contaminan ts can norm ally bearrested with high-efficiency filtration or chem ical filters, these processes can ad dsignificant costs to the maintenance of the data center

s Vibration; avoid locating the site near major sources of vibration. Airports, raillines, busy h ighways, traffic tunnels, mines and other similar sites can generatecontinu ed or intermittent vibration that could d isrupt op erations. Such vibrationcan affect disruption of utilities or support equipment, or it might directly disrupt

the hardware.s Established security; consider locating the building within an existing complex

so as to take ad vantage of established security m easures. Costs associated w iththe expansion of existing patrols and security systems will most likely be lessthan rep licating these at a separa te site. Both active and p assive secur ity measu resshould be emp loyed to protect the data center from vand alism, industrialespionage, or terrorism.

s Minimize d Target; avoid m aking the d ata center a crime target. Besides the vap or

barrier concerns norm ally associated w ith them, exterior w indow s can alsopinpoint th e location of the data center for d esigned assault. In ad dition, due toits 24 hour operation, data center window s have been known to be targeted invand alism shootings simply because they are the only lights on in the midd le ofthe night.

s Proximity to ne ighboring structures; isolate the building from the risksassociated with neighboring structures. Office areas or industrial buildings have ahigher risk of fires or other hazard s. The d ata center should not be su sceptible to

dow ntime caused by activities not associated w ith its functioning.s Emergency services; ensure ad equate access for supp ort and emergency services.

This is particularly important in congested urban areas. Access for large deliverytru cks or emergen cy fire response shou ld be free of obstacles at all times. Parkingfor emergency pow er generation or air conditioning veh icles, necessary d uring aprolonged outage, should be considered .

s Availability o f utilities ; ensure adequate utilities are available. While urban areaspose m any logistical problems, they n ormally p rovide the availability ofredu nd ant utility feeds and good infrastructure supp ort. Extreme rural areas maybe more su sceptible to single points of failure, making the d ata center m orereliant on its on-site backup equipm ent.


17/106

Chapter 1 General Site Selection Criteria 1-3

1.2 Locating the Comp uter RoomWhether a d edicated facility or part of a mu ltipu rpose building, the physicallocation of the data center is extremely important. The raised floor space, airconditioning sup port, uninterru ptable pow er sup ply (UPS), generators, and relatedsupport equipment must be coordinated with the other areas of the building andprop erly positioned w ithin the building p erimeter in order to optimize theirinteraction and the overall supp ort of operations. The location of the d ata centerwithin th e overall facility should be based on nu merous criteria, includ ing the

following general considerations.s Isolation from contaminants; isolate the comp uter room from contaminant-

prod ucing activities. Isolate the computer room from contaminant-produ cingactivities. Influences from p rint room s, machine shop s, kitchens, loading d ocks, orany area with h igh levels of contaminan t generation or op erator activity shouldbe avoided. Ensure the exhaust from generators or other sources does not d irectlyenter the intake of air hand lers serving the compu ter room.

s Access; ensure adequ ate access for h ardw are from loading dock, freight elevator

or ap prop riate entrances. This will include a pp ropriate door sizes negotiablecorners, ramp s and smooth floor sur faces. In ad dition, it is important that p roperaccess is provided in support areas to allow for service or replacement of UPS,chillers and other large items. As a facility grows or changes, access paths areoften eliminated or changed .

s Security ; provide secure p oints of entry to th e compu ter room. This helps securesensitive d ata, limit th e p ossibility of emp loyee vand alism, m inimize exposure toinapp ropriate psychrometric or contaminant conditions, and control the

possibility of failures caused by inadvertent actions of un trained personnel.s Raised flooring; design the raised floor computer spaces in convenient proximity

to the support equipment (UPS, chillers, etc.). It is often appropriate to locate thedata center on floors above the sup port equ ipment in order to consolidate coolingand power tru nklines.

s Air conditionin g; consider the type of air conditioning to be u sed. Chilled w aterunits will need to be connected to chillers located in the building or an adjoiningsup port facility, and may require cooling tow ers. Due to noise and structuralissues, chillers are norm ally located in th e basemen t of the facility or in a separa tewing of the main building. Direct expan sion air conditioners require conden serun its located outside the bu ilding. In add ition, the roof or outside p ads shou ldprovide ad equate structural stability to accommod ate the condensers.

s Risk of leaks; avoid locating the hard ware areas beneath p otential liquid leaks.Do not ru n the air conditioner piping throu gh the ceiling void of the comp uterroom. Do not locate the data center beneath kitchens, workshop s, or other areasthat h ave a h igh potential for leaks. Locating the compu ter room below buildinggrad e add s the p otential for leaks from ou tside the building. In add ition, locating

the compu ter room on the lower floors of a m ultistory bu ilding, particularly one


18/106


with m ultiple tenants, runs the r isk of leaks associated w ith a sprinkler d ischargein the floors above. Expansion joints, conduit or pipe penetrations, cracks andother breaches can all allow for water infiltration.

s Proximity to tenants; avoid locating the compu ter room n ear areas leased byother tenants. While the current ap plication of the neighboring room m ay beapp ropriate, this could change dr amatically should the lease change h and s. Inadd ition an a rea with a short-term lease may change h and s frequently,necessitating potentially disruptive renovation activity.

s Room to expand; locate the comp uter room in an area that offers the potential forfuture expansion. Even though technology changes tend to m ake hardw are morespace-efficient over time, the ability to expand, either within the current footprintof the building, or through add itions, should be available to accommod ate

possible grow th a s the room evolves. If growth is anticipated, constructingsurrounding offices on a preinstalled raised floor will facilitate the conversion tohard ware areas. If growth is not anticipated in the near future, but is still possible,app lications that can be easily moved should be considered for the surrou nd ingareas. Avoid land -locking th e compu ter room. While the expansion n eed n ot bedirectly connected to the existing areas, it is often easier to share supportequipment, such as chilled water loops or security, if they are located in closeproximity.


19/106

2-1

CHAPTER 2

General Computer Room Criteria

This Chapter d etails the proper d esign and building of a comp uter room asindicated in the Sections that follow:

s Designing th e Room on page 2-1s Designing the Floor on page 2-4s Building the Room on page 2-7

2.1 Designing the RoomProper plann ing of the data center does not end with its conception andconstruction. The comp uter environment is constantly evolving to accomm odatechanges in technology and the business landscape. Tools that help ad apt to these

changing needs are essential in a mod ern d ata center. Just as it is importan t tomonitor of environmental conditions, it is also important to keep up dated workingdrawings of the computer areas.

2.1.1 Comp uter Aided Design (CAD) Drawings

A compu terized dr afting system is an investment in the futu re of the data center.This allows for the continued up dating of the electrical, mechanical and compu tersystems. Upd ated d raw ings can be used in site evaluation and future planning, andvarious scenarios can be explored in detail. The availability of accurate, updatedplans also facilitates projects involving outside contractors. The maintenance ofupdated, computerized prints is highly recommended.


20/106


2.1.2 Design Flexibility and Planned Redund ancy

When designing the d ata center, it is imp ortant to includ e ad ditional resources for

redu nd ancy. This may be in th e form of available pow er, environmental supp ortequipment or floor space. This redundancy allows for the flexibility necessary toaccomm odate changes and short-term growth associated w ith hardw are upgra des. Italso allows for uninterrupted operations during upgrades or replacements inhardw are. New hardw are can be run simultaneously w ith the hardware it isreplacing, rather than swap ping the tw o. Redu nd ancy also allows security in theevent of a failure. This is par ticularly tru e of the environmental sup port equ ipment.

While most data centers are designed w ith at least a minimal amou nt of redund ancy,

this issue is often forgotten in future plann ing. Excess floor space or sup por t systemsthat w ere designed for redu nd ancy, are often used for growth, reducing theprotection they once provided . It is imp ortant to carry throu gh this impor tant factorin the futu re planning of the room. The redu nd ancy must be maintained, even as thedemands of the data center grow. The amount of redundancy planned can beincreased in th e design p hases to add ress this. This will provide room for grow thwh ile still providing the back-up needed . If this is not d one, the supp ort systemsshould be increased along w ith the hardw are du ring the expansions. Failure to

provide adequate redundancy can lead to logistical problems and may degrade theoverall reliability of th e comp uter operations.

2.1.3 Expansion Considerations and Preparations

Each evolution in hard wa re technology dr amatically increases the comp uting ordata storage capacity per square foot of the room. This is how m ost data centershave been a ble to survive for so long w ithout continually expan ding th eir physicaldimensions. The evolution of the computer room is normally a continuous process ofminor grow th and changes within a larger cyclical pattern of more dram aticchanges. Ha rdw are will normally continue to grow in the room u ntil it nearscapacity, then changes will be m ade to regain som e of this floor space throu ghup grades in technology. The compu ter room w ill then begin to encroach on thisnew ly available space as dem and s on the compu ter room continue to increase, andthe cycle will repeat. In some cases the changes in technology and the increased

needs for compu ting or storage capacity w ill evolve at the sam e speed, in otherinstances, one will out-pace the other.

When technology evolves more quickly than the need s of the business, the d atacenter will normally develop open a reas, devoid of hardw are. These may rem ain forsome time, and are often very attractive real estate to other areas of the bu siness. Itis imp ortant not to be too quick to dow n-size the d ata center areas, as prop erlydesigned hard ware spaces are much m ore expensive to construct than typ ical office

environments. All planning of expan sions or red uctions to the raised floor compu ter


21/106

Chapter 2 General Computer Room Criteria 2-3

room areas m ust be considered in ma cro terms. It is often m uch m ore financiallypru dent to allow portions of a well-designed room to remain vacant than to try toreconstruct this from converted office space when the d ata center d emand s increase.

If the needs of the bu siness outpace the evolution of the technology, it may benecessary to increase the physical dimensions of the data center. This should also bedone in conjunction with long-term planning. Moving to more space efficienttechnologies, wh en ava ilable, may p rove to be more cost effective tha n expan dingthe ph ysical dimensions of the comp uter room .

2.1.4 Room Layout and PlanningWhether a d esign-built room, or a renovated area, the comp uter room mu st be ableto accomm odate d iverse hardw are designs and requirements. The mission of acompu ter room rarely remains stable, and the hard ware d esigns and configurationschange as technology and th e goals of the compan y evolve. While the future of acompu ter room can r arely be anticipated, it is essential that th e hard ware a reas areplanned in such a fashion as to allow for seamless adap tation to the changing needs.

The main criteria driving the type of hardw are in the room will be determined bythe applications of the business. The following general guidelines should be used inplanning the current layout of the Sun Microsystems hardware and supportequipm ent, and to help in prep aration for future changes.

s Do not d etermine air conditioner placement based on convenience. Often airconditioners are placed aroun d the perimeter of the room because of theconvenience of piping, water detection an d other issues. This is not norm ally themost effective placement for the units, except in some smaller rooms. The

hard ware h eat-load of the room w ill change repeatedly over the life of the room,and it is important that the p rimary criteria in the d etermination of the airconditioner placement be its effectiveness in addressing the current planned load,and their ad aptability to changes in configurations.

s Consider the air-flow p atterns of the hard ware being installed. Does it d rawsup ply air d irectly from th e subfloor? Does it dr aw air from the am bient roomabove the floor surface? Is the air exhausted out the back or top of the cabinet? Ifthe hard ware is not in a cabinet, or if it is in an open rack system, w hat is the

design of the individual components? Does air flow side-to-side, front-to back,front-to-top or bottom-to-top? Be sure th at the un its are not laid ou t in a fashionthat exhausts air from on e un it into the intake of the next.

s Provide ad equate aisle space to allow for un obstructed passage, and to allow forthe replacement of cabinets within a string withou t interference to n eighboringun its. Strings of cabinets should be kept to ma nageable lengths so as to allowclear passage between aisles in the event of an emergency, or in order to respondto a problem with a unit. Long, unbroken strings necessitate a significant amount

of time to m ove from one aisle to another, or even from the front of a un it to itsback.


22/106


s Design the operator traffic patterns to minimize the possibility of accidentalcontact with sensitive components. Avoid placing operations in a fashion thatnecessitates personnel trav eling th rough a sensitive area, such as th e m ainmachine room, to a less sensitive area. For example, operators should not have to

travel through the compu ter areas to get from the comm and center to the printroom.

s Position hardware in strings or rows that run perpendicular to the air handlers.This allows for an u nobstructed retur n of heated air back to the air conditioners.Where possible, avoid air-flow patterns that necessitate the air traveling over tallhard ware cabinets to return to the air conditioners.

2.1.5 Computer Room Access

Access to the computer rooms should be strictly regulated, and limited to only thosepersonnel necessary for its operation. All personnel w orking within th e data centershould have at least a basic und erstanding of the sensitivities of the hard ware so asto avoid activities that p ose a d irect risk to the hard ware. Accidental contact w ithhard ware buttons, cable connections, terminals or em ergency response controls can

all cause system interrup tions of varying degrees.All points of access to the comp uter room s and other sensitive areas should becontrolled by checkpoints or coded card readers to restrict access to authorizedpersonnel. Security personnel should also remotely m onitor points of entry v iacamera.

2.2 Designing the FloorThe raised access floor system provides the flexibility in wiring, hardware locationand air cond itioning. The raised floor shou ld be constru cted of 24 inches x 24 inches(61 cm x 61 cm) panels interchangeable with perforated tiles for air distribution orcustom cut tiles for cable or utility passage. This design isolates data lines, powercables and piping to p rovide a safe environmen t for operators and to protecthard ware op erations. In add ition, the raised floor design provides a mean s forflexible and efficient air distribution to the hardware. While it is possible toaccomm odate a small num ber of hardw are units in alternately designed rooms, it ishighly recomm ended that large num bers of hardw are cabinets are installed in araised floor system.


23/106


2.2.1 Floor Height

There should ideally be 24 inches (61 cm) between the raised floor system and the

structural d eck. A m inimum of 18 inches (46 cm) shou ld be provid ed. Variationsfrom this figure should be based on air conditioner d esign an d anticipated subfloorcongestion. Inadequate subfloor depth will lead to difficulties in systemsreconfiguration over time, may make the removal of u nused or obsolete cablesdifficult, and will likely obstruct airflow. Additional space may be advisable ifsubfloor obstructions are abu nd ant.

2.2.2 General Sup port Grid RecommendationsA raised floor system u tilizing bolted stringers is recomm ended to provide themaximum rigidity for d ynam ic loads as w ell as to enhan ce the signal reference grid.Snap-on stringers often come loose affecting the integrity of the floor structure.While some stringerless systems claim the same benefits as bolted stringer systems,a great d eal of research shou ld be d one pr ior to choosing an alternative to therecomm enda tion. In add ition, stringerless systems may requ ire additional supp orts

wh ere custom cable cutou ts are m ade for air or cable transfer. In ad dition, boltedstringer systems allow for the removal of ad jacent tiles withou t threat to theintegrity of the floor.

2.2.3 General Tile Construction Recommendations

The floor tiles in th e raised floor shou ld be 24 inch x 24 inch (61 cm x 61 cm). The tilecore may be constructed of compressed w ood or concrete, or may be an op enstructural m etal design. The entire tile should be constructed of, or encased in,galvanized or painted steel. Alternately, cast alum inum tiles may be used .

The tiles should have a high-pressure laminate top surface. The floor surface m ustallow for proper dissipation of electrostatic charges. The floor tiles and grid systemsshould p rovide a safe path to groun d th rough the tile surface, to the floorsubstructure an d through the signal reference grid. The top sur face of the floor

covering to u nd erstructure resistance should be betw een a m inimum of 1.5 x 105ohms and a maximum of 2 x 1010 ohms (as per NFPA 56A Test Method). The panelstructure (not sur face laminate) to un derstru cture resistance should be less than 10ohms.

Carpeted tiles should not be used in hard ware areas. Carpeted tiles can har borcontaminants that are agitated every time th e tile is walked on . In ad dition, thesetiles are more easily damaged by the movemen t of hard ware, or even wh en removedusing specially designed tile lifters that incorporate spikes designed to catch theloops of the tiles. Carpeted tiles designed with static dissipative prop erties can


24/106


become less effective in this regard as they w ear over tim e. Carpeted tiles that arelaid over an existing raised floor surface, are normally offset from the grid, andmake it more difficult to access the subfloor. Carpeted tiles should only be used incomman d centers, or other that do not hou se sensitive hardw are, and do not require

frequent access to the subfloor void.

2.2.4 Floor Maintenance

The tile surface shou ld be maintained to the man ufacturer's specifications. Theguidelines listed in Section 4.2.6, Cleaning Activity on page 4-5 should be used.

No w axes or insulative coatings should be used , as these can form a barrier thatinterferes with the static dissipative properties of the floor. It is also extremelyimportan t that the stringers and p edestal tops be kept clean, as a buildup ofcontaminants on these surfaces can potentially impact the functioning of the floor aswell. Replace tiles as the surface becomes dam aged, or as they become w arped byheavy loads. A dam aged su rface or a tile that does not sit tightly in the grid canaffect the ability of the tile to properly dissipate static charges, and could pose asafety hazard.

2.2.5 Cutouts and Other Tile Customizations

Tiles will need to be customized to accomm odate the shape of the room , the airconditioners and the hardware. All tile modifications should be performed accordingto manufacturer recommendations. Additional structural support may be necessarywh ere partial tiles are installed along w alls, around colum ns or by a ir conditioners.

The exposed cut edges of all cut-outs for cable or air passage should be capped withprotective trim. Exposed metal ed ges can dam age cabling, and the exposed cores ofsome tiles can shed particulate matter into the airstream.

2.2.6 General Load Rating Recommendations

The load capacity of the structural floor mu st be taken into account wh en d esigningthe data center. Some areas designed for light duty, such as office, may beinadequ ate. A qualified structural engineer shou ld be consulted in the evaluation ofpotential areas for the location of a new data center within an existing building.Enhanced sup port m ay be adv isable in high traffic areas, or areas with h eavier thannorm al loads. Enhanced supp ort should also be considered for ramp s and th e raisedfloor areas imm ediately above them.


25/106


The raised floor load rating w ill vary dep ending on the design and use of the room.In most cases, a floor designed for a concentrated load imp osed by stationaryfurn iture an d eq uip men t of 1000 Lbs (454 kg) with a m aximu m d eflection 0.080 inch(0.2 cm) from any point on panel top should be sufficient. Rooms or areas with high

levels of motorized traffic or heavy rolling loads should consider a higher rated tile.

2.2.7 Fire Rating

The raised floor system should be in compliance with the specifications laid out inthe N ational Fire Protection Associations Docum ent, N FPA 75: Protection of

Electronic/ Data Processing Equipment within the USA, or relevant N ationalstandard s outside of the USA.

2.2.8 Supp lemental Bracing

While the pra ctice should be avoided wh en p ossible, it is sometimes n ecessary tolocate data centers in seism ically active zones. Seismic bracing for th e raised floorsystem can normally be obtained from the floor manufacturer. As a general practice,heavier components shou ld be installed lower on the racks to avoid top-heavyequipment.

2.3 Building the RoomNo activity in the d ata center should be allowed to significantly degrad e theenvironment. Because of the dynamic nature of a data center, it is often necessary toimplement projects to add ress its changing needs or mission. This may encompassmoving w alls to expand or redu ce the size of the compu ter space, replacing olderfloors or ceilings, or up grading en vironmental sup port equ ipment, amon g otherthings. It is essential that these actions, meant to improve the stability and operation

of the data center, are not allowed to degrad e conditions and threaten up time.Precautions m ust be taken to control psychrometrics and air distribution, and tolimit or contain contam inant p rodu ction. Even though the actual activities arecommon, the environmental requirements of the comp uter room p ose uniqueproblems. Norm al cutting, dr illing or dem olition is unacceptable withou t prop erprecautions.


26/106


2.3.1 Building Preparation

Whether the d ata center is located in a new or existing structure, the building m ust

be prop erly prepared to receive the hard ware prior to its installation. Constructionprojects are expensive, and the construction of controlled data center areas is moreexpensive than most. There is always a great d eal of pressure to m eet dead lines andkeep w ithin costs, but it is extremely impor tant that this is not achieved by cuttingcorners and settling for inferior w orkman ship.

A d ata center requires more precise control over tem peratu re, relative hum idity,airflow, and contaminants than does a typical office environment. If the specificneeds of this environment are to be met, they must be add ressed th roughou t the

design and construction of the room. Prior to installation of the hard wa re, the roomshould be inspected to identify any remaining exposures, and all surfaces in theroom m ust be ap prop riately d econtam inated. The final pun ch-list items shouldinclud e specific tasks designed to bring the comp uter room environment withinspecific param eters. The following a ctions shou ld be performed in the gen eral orderlisted.

s Perform a general construction decontamination of the room. A rough cleaning ofthe room should first be condu cted to remove all ma jor construction-related

debris. Low-grad e indu strial vacuum s can be used at th is stage to remove heavydep osits. This stage of the cleaning w ould includ e those steps typ ical to anyconstruction p roject, and is aimed p rimarily at app earance.

s The vapor bar rier of the room should be checked. Any breaches that could allowsignificant m oisture m igration shou ld be n oted an d fixed. In th e subfloor void,these can include p erimeter penetrations around pipe or condu it passages, cracksin the d eck or p erimeter wa lls, expansion joints or open d ucts or w alls thatconnect the subfloor void to the ceiling void or to other floors. In the above floor

space, these can include h oles or cracks in the p erimeter w alls, gaps around pipeor du ct penetrations, interior or exterior w indow s, access wind ows or p erimetermail slots, gaps around doors or light fixtures w ith designed a ir vents. Above thedrop ceiling, breaches can includ e gaps arou nd pipes, du cts or condu it. Gapsaround structural beams or between perimeter walls and corrugated roofingmaterials, open ceiling voids to other areas, gaps a round access doors to th e roofor other floors or roof vents.

s Load test the generators, UPS and other pow er infrastructure comp onents.

Perform functionality tests to ensure the d ata center is ready to accomm odate on-line compu ter operations.s Perform a final inspection on the environmen tal supp ort equipm ent. Check for

proper installation an d functioning of all equipm ent. Put the air conditioners andhum idifiers through their cycles by adjusting th eir set points to force them intostages of cooling, hea ting, hu mid ifying, d ehu mid ifying, etc. By this stage, chillers,UPS, generators and all other similar sup port u nits should ha ve been tested.Make any n ecessary adjustments.


27/106


s Prior to the installation of the hardw are, but du ring and after the constructioncleaning, the air conditioners should be run continuou sly to filter the room air.These units need n ot be cooling, as the primar y pu rpose of this action ispa rticulate ar restance. Ideally 60% efficiency filters w ill be used at th is point.

These filters will be replaced prior to hardware installation, as they will likelybecome inun dated with p articles that can be re-dispersed by the subfloor pressureforcing air through a un it in a reverse pattern should on e of the air conditionersbe turn ed off.

s After the construction cleaning, it is essential that a thorough decontamination ofthe room surfaces be condu cted to remove any residu al particulate in finalprep aration for the installation of the hard wa re. Low-grade vacuum equipm ent,such as that u sed d ur ing the constru ction phases of the project, lacks the filtration

necessary to arrest most particulate. Vacuums equipped with High EfficiencyParticulate Arrestance (HEPA) filtration mu st be u sed a t this stage. In ad dition,special equipment an d p rocedu res should be emp loyed, as outlined in the latersections of this report.

s Prior to the hard ware installation, the room should be stabilized w ithin the goalspecifications established within this document. It may be difficult to achieveprecise balancing and app ropriate environmental sup port equ ipment cycling inthe controlled space until the d esigned heatload is installed, but conditions

should be m ade as balanced an d a pp ropriate as possible prior to installation.Temperatures, relative humidity levels, subfloor pressurization, roompressurization and airborne par ticulate levels should all be checked.

2.3.2 Build ing Materials Selection

All building m aterials should be chosen with an aim tow ards cleanliness andmoisture retention. Materials should be chosen or treated to avoid sh edd ing ordeterioration that m ight be tolerable in m ore loosely controlled en vironments.Particular attention shou ld be p aid to areas in th e direct airflow p atterns of theroom, and materials that require repeated movement or disturbance in the normaloccupation of the room. Ceiling tiles should have a vinyl or foil face that willprovide a moisture barrier, and will help protect the tiles from shed ding as th ey aremoved . All supp ly plenum surfaces should be constructed of app ropriately treatedmaterials, such as encapsulated concrete or galvanized or p ainted m etals.

Ideally, mater ials in keeping w ith th e design of a class 100,000 cleanroom sh ould beconsidered appropriate.

Panels and other necessary items should be p re-cut and drilled outside the room tominimize the activity necessary w ithin the room. This may ad d time and effort, butwill help limit contaminan t prod uction within the room.

Not all activity can be per formed outsid e the room, so it is also imp ortan t that efforts

are mad e to contain or arrest contaminan ts produ ced by activities performed w ithinthe controlled space. Plastic sheeting can be used to isolate work areas from other


28/106


areas of the room. Portable filter systems can be rented or p urchased to h elp arrestparticulate in the a ir (it shou ld be n oted th at these are n ormally only effective inlocalized areas). Vacuum units equipped with High Efficiency Particulate Air(HEPA) filtration should be used to ad dress any contamination p rodu ced by dr illing

or sawing as soon as it is produ ced.

In add ition, it is extremely important th at the temp erature, relative hum idity and a irdistribution conditions are taken into account, and that these conditions are notsignificantly degraded . Doors to the data center mu st not be left open, it may benecessary to design a temp orary p ersonnel trap to limit exposure caused byincreased traffic through the controlled spaces. Similar isolation may be necessary ifchanges to the room perimeter p rodu ce exposures. The job progression shou ld be

planned in such a mann er so as to limit exposures. Care should be taken wh enremoving floor tiles to ensure that the subfloor p ressure levels remain adequ ate forprop er air d istribution. It is also imp ortant th at any activity that involves theenvironmental sup port equ ipment be carried out w ithout affecting the ability of theun its to add ress the conditioning and hu midification need s of the subject areas.

Proper imp lementation of construction and renovation projects in an on-line datacenter require additional time, planning an d expense, but these precautions areessential if the un interru pted oper ation of the da ta center is a priority. Ignoring these

issues can lead to catastrophic failures, or long-term p erformance problems.


29/106

3-1

CHAPTER 3

Cooling and Air Distribution

Maintenance of app ropriate temperatu re and relative hum idity (RH) levels in a datacenter environm ent is extremely important. Comp uter h ardw are requires a balancedand app ropriate environment for optimal systems operations. Temp eratures or RHlevels outside the specified operating ranges, or extreme swings in conditions, candet rimen tally affect compon ent reliability. To comp licate th is, the heat load of mostcompu ter rooms is extremely complex and experiences normal fluctuations d uringits evolution. A great deal of importance mu st be placed on maintaining app ropriate

temp erature and RH levels through out the d ata center at all times. The informationin this Chapter includes the following Sections:

s Temp erature Recommend ations on p age 3-2s Relative Hu midity Recomm enda tions on p age 3-2s Section 3.3, Temperature and Relative Humidity Problems on page 3-3s Temp erature an d Relative Hu midity Monitoring on p age 3-5s Air Conditioner and Hu midifier Design, Maintenance, and Placement on

page 3-6s Mechanical Support Systems on page 3-10s Air Distribu tion Tile Placement on p age 3-10s Ha rdw are Placement on page 3-13s Subfloor Pressure Differential Recommendations on page 3-17s Sup ply Air Plenum Integrity Concerns on page 3-18s Vapor Barrier Design and Conditions on page 3-19


30/106


3.1 Temp erature Recomm endationsAn am bient tem peratu re ran ge of 70 to 74 F (21 C to 23 C) is optimal for systemreliability and operator comfort levels. Most computer equipment can operate withina w ide p sychrometric range, bu t a tempera ture level near 72 F (22 C) is d esirablebecause it is easier to maintain safe associated relative humidity levels at thistemp erature, and there is an acceptably w ide operational buffer in case ofenvironmental support systems down-time. Though individual standards varyslightly, 70 to 74 F (21 C to 23 C) shou ld be u sed a s an op timal recom men da tion.

3.2 Relative Hum idity RecommendationsRelative hum idity is the amou nt of moisture in a given sam ple of air at a giventemp erature in relation to the maximum amou nt of moisture that a sam ple could

contain at the sam e temperatu re. A volum e of air at a given temp erature can hold acertain a mou nt of m oisture. Because air is a gas, it expan ds a s it is heated. As airgets war mer, its volume increases and th e amou nt of moisture it can hold increases,thus causing its relative humidity to decrease. Therefore, in a system utilizingsubfloor air d istribution, the ambient relative hum idity will always be lower than inthe subfloor.

Am bient relative hum idity levels between 45% and 50% RH are the most su itable forsafe data processing operations. Und er certain circum stances, most d ata p rocessingequipm ent can op erate w ithin a fairly wide en vironmental ran ge (20% to 80% RH),but the optimal goal should be specified at 45% to 50% RH for several reasons. Theoptimal ran ge helps p rotect compu ter systems from corrosivity p roblems associatedwith high hu midity levels. It also provides the greatest operating time bu ffer in theevent of environmental control system failure. In ad dition, this range helps avoidfailures or temp orary m alfunctions caused by intermittent interference from staticdischarges that occur when relative humidity is too low. Electrostatic discharge(ESD) is easily generated and less easily dissipated in areas where the relative

humidity is below 35% RH, and becomes critical when levels drop below 30% RH.The 5% RH r ange may seem u nreasonably tight w hen compared to the guidelinesused in typical office environments or other loosely controlled areas, but it is not sod ifficult to mainta in in a data center b ecause of the high efficiency vap or barr ier andlow rate of air changes norm ally present


31/106

Chapter 3 Cooling and Air Distribution 3-3

Though the d esign limits for this hardw are are wide, conditions should always be

maintained near th e optimal levels. It should also be noted that certain extremeswithin th is range can lead to u nacceptable conditions. If very high temp eratures aremaintained in conjunction w ith very high relative hum idity cond itions,condensation can occur. Also, if very low temp eratures are m aintained w ith verylow relative hum idity levels, even a slight rise in tem peratu re can lead tounacceptably low relative humidity levels. The reliability and life expectancy of thehard wa re will be enhanced by m aintaining conditions within the optimal ranges.

3.3 Temp erature and Relative Hu mid ityProblemsTemp erature an d relative hum idity conditions should be m aintained at levels that

allow for the greatest operational buffer in case of environmental su pp ort equipm entdow n-time. The goal levels for the comp uter room should be d etermined in aman ner th at w ill achieve the greatest op erational buffer and the least possibility ofnegat ive influence. The specific hard ware design, room configur ation, environm entalsupp ort equipm ent design and other influencing factors should be taken intoconsideration w hen d etermining the sp ecific relative hum idity control app ropriatefor a particular room. Psychrometrics can affect hardw are through therma linfluences, Electrostatic Discharg e (ESD), and increases in en vironm ental corrosivity.

TABLE 3-1 Environmental Requirements

Environmental

Factor Optimal Operating Non-operating

Temperature 70 to 74 F (21 to 23 C) 50 to 90 F (10 to 32 C)*

* Severe temperatu re or relative hum idity swings should be avoided . Conditions should not be allowed to change by more than 10F (5.5 C) or 10% RH in an y 60 minute p eriod of op eration.

-4 to 140 F (-20 to 60 C)

Relative H um id ity 45% to 50% RH 20% to 80% RH 1 up to 93% RH

(noncondensing)

Altitude

For altitud es outside these ranges please consult your Sun Microsystems representative.

up to 10,000 ft (3,048 m) up to 10,000 ft (3,048 m) up to 40,000 ft (12,192 m)


32/106


3.3.1 Thermal Concerns

Temp erature conditions should be maintained as close to the optimalrecomm endation as p ossible. Near the limits of the temp erature ran ge, there is lessof a buffer to absorb influences from actions in the room. The temperature profilecan be dram atically affected by p roblems w ith the environmental sup portequipm ent, such as component failure or even schedu led preventive maintenance.

There is also less tolerance w hen th e heat load of the room changes du e to theinstallations, de-installation or reconfiguration of hardware. Significant changes incooling tem peratu re can also be caused by the rem oval of excessive nu mb ers of floortiles for subfloor w ork, such as cabling. App ropriate temp eratures also ma ke it

easier to m aintain app ropriate moisture levels. Even if app ropriate relative hu miditylevels can be maintained at high or low temp eratures, it is likely that the actualmoisture content, in grains of moisture, of the air wou ld be inap propr iate.

3.3.2 Electrostatic Discharge (ESD)

Another consideration is the generation and neutralization of static charges, as wellas the effects such charges can have on sensitive electronics equipment, as well assome mean s of preventing such dam age. Appropr iate relative hum idity levels helpmaintain a n env ironment in wh ich static charges are more easily d issipated. Pleaserefer to Section 6.8 for additional information.

3.3.3 Corrosion

High hum idity environments increase the corrosivity potential of gaseouscontaminants in the atmosph ere. Gases suspen ded in the air can be transferred to theroom's hard wa re via the moisture in the air. It should a lso be noted th at d rasticchanges in temp erature can cause latent heat chan ges, leading to the formation ofcondensation. This is particularly common in areas wh ere hot and cold air meet.Excessive moisture in the air and condensation can cause a nu mber of hard wareproblems. Water can react directly w ith metals, attacking them and forming

corrosion. Another w ay that a m oist atmosph ere can d egrade ha rdw are is byforming cond uctive solutions (electrochem ically). If there are electrical potentialdifferences between two dissimilar metals in a component, electrolytic or galvaniccorrosion processes can set in. The water can also form a more reactive combinationwith gases present in the atmosph ere. The resultant compou nd s can then attack thehardware surfaces. Excessively high RH levels should be avoided.


33/106


3.4 Temp erature and Relative Hu mid ity

MonitoringAccurate and comprehensive monitoring of environmental support equipment andin-room cond itions is extremely importan t in an environment as complex andsensitive as a data center. The monitoring system used must effectively assess theroom conditions, or it w ill provide an inaccurate representation that can lead toinapp ropriate actions or ill-found ed assump tions. The following considerationsshould be addressed

s The system in place mu st provide a detailed and representative profile of roomconditions. If a single point of reference is used, it will not give an accuratepicture of the room's profile. If a single sensor is placed in an area withapp ropriate conditions, such as on a colum n d irectly above a perforated tile, themonitoring system wou ld be indicating that room cond itions are app ropriateeven though this may not be the case. Assum ptions concerning the environm entthat are based on such data can lead to decisions that could actually degradeconditions. The sam e can be said about a m ulti-point system tha t hasinapp ropriately placed sensors.

s The system should have h istorical trend capabilities. The d ata gleaned fromanalysis of historical p sychrometric information can be instrum ental indeterm ining seasonal changes or other ou tside influences. The d ata should beeasily available, and the operating system should be p owerful and adap table.

s The system should have critical alarm capabilities. At the very least, the systemshould be set to notify app ropriate personnel wh en conditions move outsidecertain p aram eters. Depend ing on the design of the da ta center, it m ay also be

useful to have a system th at p erforms certain fun ctions au tomatically, such asswitching to a back-up chiller if the primary chiller fails.

s The system configuration an d d ata should be periodically examined andevaluated by trained p ersonnel to ensure that they are app ropriate for the currentroom d emand s, and to identify any problems missed in the day-to-day operationsof the room.

s Ideally, an integrated bu ilding m onitoring system shou ld be u sed to trackconditions in all of the bu ilding system s. This wou ld includ e not only the in-room

air conditioners and hum idifiers, but also the cooling sup port systems, powerback-up , fire detection an d sup pression, water d etection, security and otherbuilding infrastructure and life-safety systems.

While an ad ded expense to the d esign an d maintenance of a facility, comp rehensivemonitoring systems provide an invaluable tool to building maintenan ce personnel.They are essential in correcting current problems in an expedient ma nner andidentifying p otential susceptibilities before they im pact h ardw are op erations.


34/106


3.5 Air Cond itioner and Hum id ifier Design,

Maintenance, and PlacementWhile the specific details of the environmental su pp ort systems w ill vary from site-to-site, an air conditioning system w ith a dow nw ard flow should be utilized in alldesigned comp uter rooms. App ropriate cond itions can be maintained for a smallamou nt of hardw are space utilizing other d esigns, but the air-flow pattern sassociated w ith the dow nw ard flow d esign allow for the most efficient cooling of thehard ware. The data center should not be designed based on th e same school ofthought used in general comfort cooling applications. The efficiency of a precisionair conditioning system is not only measured by the d egree of temp erature control,but a lso on the ability of the system to get the conditioned air to th e un its ofhard ware in need of cooling. Compu ter rooms require precision cooling for anu mber of reasons:

s The heat load in a compu ter room is very dense. Compu ter rooms generally havesix to eight times the heat density of normal offices.

s The heat load in a compu ter room va ries across the room's area. The airconditioning system m ust be able to add ress the specific needs of particular un itsof heat-prod ucing hard ware in order to achieve a balanced p sychrometric profile.

s A compu ter room requires precision tem peratu re control. The air conditionersneed to be set accura tely with a sensitivity of +/ - 2 F (+/ -1 C) and +/ - 3% RH orcloser.

s Electronic equipment rad iates a "drier" heat than the hu man body. Therefore,compu ter room precision cooling systems requ ire a higher sensible heat ratio(SHR) than office areas. Ideally, a computer room cooling system should have a

sensible heat ratio of 1:1; that is, it shou ld p rovide 100% sensible cooling. Mostprecision systems have between 85% and 100% sensible cooling, while comfortcooling systems n ormally rate m uch lower.

s The comp uter room heat load changes w ith add itions or reconfiguration ofhard ware. The air conditioning system mu st adap t to these changes.

s The precision cooling environment m ust p rovide an ad equate change of air in theconditioned space. While a normal office cooling environment requires only twoair changes per hou r, the high-density heat load in a compu ter room requ ires as

man y as 30 changes p er hou r. Precision air cond itioners p ass more th an 500CFM/ ton, comfort cooling a ir cond itioners m ay p ass as little as 350 CFM/ ton. Ifthe volum e of air is inad equate, the sup ply air w ill heat up before it reaches theareas in need of cooling, and will subsequently be less effective in addressing theheat loads in the room.

The dow nw ard-flow air conditioning system typ ically u tilized in d ata centers andother similar environments incorporates a raised floor system. The raised floorshould be 24 inches (60cm) with a minimum of 18 inches (47 cm) above thestructural deck to allow for both the runn ing of cables and p ipes as well as for the

di ib i f di i d i h h d Th d l il d i ll f


35/106


distribution of conditioned air to the hard wa re. The m odu lar tile design allows forease of movem ent for both hard ware an d air d istribution reconfigurations. Ashard wa re is add ed, solid tiles are replaced by tiles with cut-outs to allow cableaccess, and perforated tiles are positioned to deliver conditioned air to the hardware

intakes.

In the general airflow patterns associated with most computer room process coolers,room air en ters the top of the air conditioner (Return Air) where it is cleaned by airfilter banks. As the return air passes over th e cooling coil, the air temp erature islowered significantly, and large fans at the bottom of the un it pu sh the conditionedair (Sup ply Air) into the room's subfloor void w here it is introdu ced into thecomputer space via cable cutouts, floor grilles or perforated floor tiles. Once in theambient room space, the cond itioned air m ixes with the hard ware heat load an dflows back to the air conditioners for reconditioning. This produces an efficientairflow p attern.

FIGURE 3-1 Typical Upward Verses Downward Computer Room Air Flow Patterns

Centralized systems, using a single large air hand ling u nit, should be avoided.Whether the system introduces the conditioned air into the subfloor, or directly intothe hard wa re space, centralized systems generally lack the redu nd ancy present withindividua l packaged units. Centralized systems also lack the d egree of controlpresent w ith mu ltiple units. In m ost centralized systems, the temp erature andrelative hu midity are regu lated by a single sensor set in the ambient sp ace of the

i th t i d t Thi i i t t ti f


36/106


room, or in the retu rn a ir du ct. This can give an inaccurate representation of roomcond itions, as it is un likely that cond itions in all areas are the sam e as they are at th issingle sensor p oint.

Overhead air introduction or upflow air cond itioning should be avoided du e to theirassociated tur bulent airflow patterns. The majority of the hard wa re in most roomstakes in air for cooling at th e front or bottom of the u nit ad exhausts it out the backor top . Introducing conditioned air from the ceiling level causes turbulence as theconditioned air meets the hot exhaust.

It is extremely importan t that the h ardw are in the room be taken into considerationin order to ensu re that the air conditioners have adequ ate control over cooling andheating cycles. Hard ware un its that rely on direct subfloor cooling are p articularly

susceptible to large swings in temp erature associated with direct expan sion airconditioners that have only two or four stages of cooling. Because the regulation ofthe air conditioners is controlled by sensors in the return airflow, there is a delay inresponse that can cause significant fluctuations in conditions. Chilled water systemswith m odu lated controls are more approp riate where temp erature fluctuations are asignificant concern .

Hu midification can take place within th e in-room packaged air conditioners, or it

can be provided by stand -alone units mou nted w ithin the room. While ordering thehu midifiers as a feature of the air conditioner may be m ore convenient, a moistureintrodu ction d irectly into the war mer ambient air is preferable to an introdu ctioninto the cold supply air within the enclosed air conditioner. The followingconsiderations should be taken into account w hen d etermining the d esign of thesystem.

s Ideally, the actual p rocess of hum idification should not take place within th eroom's air conditioners since cold air flows cannot accept high levels of moisture.

Condensation formations and wa ter spills are a common occurrence withinprocess coolers, and can cause accelerated levels of internal corrosion. Exposingthe air conditioner's internal surfaces to excessive moisture levels will greatlyredu ce the operational life cycle of the units an d contribute to the d ecay of thecomputer operating environment.

s The subfloor void shou ld not be u sed as a m eans to distribute moisture since thisarea norm ally contains cold air that may already be near saturation. Localcondensation and corrosion on u ntreated m etal surfaces can result. The bestmethod of introducing moisture into a compu ter room is to inject it d irectly intothe ambient air space wh ere it will mix easily w ith the ambient temp eratures.

s The actual process of generating moisture should n ot prod uce unw antedcontaminants su ch as m ineral salts or other crystalline p articles.

s The hum idification system should be responsive to room conditions andequipm ent d emand s. Hardw are reliability is enhanced if psychrometric rates ofchange are kept to a nar row m argin. Hum idification systems that can monitorand adap t to the ever-changing data p rocessing environmen t are most desirable.

One recommen ded hu midification system d esign u tilizes a closed water bottle that


37/106


One recommen ded hu midification system d esign u tilizes a closed water bottle thatcontains electrodes to qu ickly heat the contained w ater and prod uce steam. Theseun its can be installed within the actual room space as w ell as inside the airconditioners. The closed bottle design rem oves any su spend ed particles from the

supp ly water, resulting in a clean m oisture su pp ly. If properly d esigned andmaintained, alternate designs, such as u ltrasonic humidifiers, can be u tilizedeffectively. Building steam systems distributed by a central air conditioner shouldnot be used. These do not have the flexibility available from multiple in-room units.As w ith the air cond itioning, the use of m ultiple hum idifiers provides localizedcontrol or red und ancy.

3.5.1 Air Cond itioner and Hu midifier Set-pointsThe set points of the environmental sup port equ ipment m ay vary between sites, andeven between ind ividual un its at the same site. One of the advantages of mu ltiplepackaged un its is the ability to mod ify set-points in localized areas. The hard wa reheatload in a room may vary from dense placement of hardware with high heatgeneration to open areas w ith little or no hardw are. While the main mean s ofadd ressing su ch variances is w ith adjustments to th e air d istribution tile placement,minor ad justmen ts to the air conditioner or hu midifier set-points m ay also benecessary.

Under most circumstances, air conditioners should be set at 72 F (22 C) with asensitivity ran ge of +/ - 2 F (+/ -1 C). Hum idifiers, in m ost cases, should be set at48% RH with a sensitivity ran ge of +/ - 3% RH. The set-points of th e air cond itionersshould alw ays be chosen in an effort to maintain the optimal recommend edtemp erature and relative hu midity levels for the room environment. These set points

should m aintain app ropriate conditions, while allowing w ide enough sensitivityran ges to help avoid frequen t cycling of the u nits. While these tight ranges w ould b edifficult to ma intain in a loosely controlled office environm ent, they sh ould be easilyattained in a controlled data center.

Nu merous factors, such as heat-load and vapor barrier integrity, will influence theactual set-points. If the room lacks adequate vapor barrier protection, for instance, itmay be necessary to ad just h um idifier set points to accomm odate seasonalinfluences. Ideally, all inapprop riate influences on the d ata center environment will

be eliminated, but in the event that they are n ot, minor adjustments, made bytrained personnel, can help alleviate their effects on the environment.


38/106


3.6 Mechanical Sup port SystemsThe specific design of the mechanical sup port systems w ill be depend ent up on thetype of cooling system chosen. Nu merous factors w ill go into the d ecision as towh ether to u se direct expansion compressorized air conditioners or a chilled w atersystem. The specific requirements of the hardware to be cooled should be taken intoconsideration w hen d etermining the type of air cond itioning system.

The availability of space to house large chillers, cooling towers, and piping systemsassociated w ith chilled w ater d esigns m ust be considered. Direct expansion u nits

require groun d-level or roof space to hou se large condensers. The climate of theenvironment in which the building is located is also a governing factor, as certaind esigns m ay be easier or less expensive in certain clima tes. Issues of flexibility andredu nd ancy should also be evaluated . The system installed shou ld provide ad equateflexibility for growth, and 100% redundancy. The loss of any one component of thesystem should not hav e a significant imp act on the system as a wh ole. The systemshould be d esigned in such a fashion so as to accomm odate repairs or changes w hilethe system is on-line, or w ith minimal dow ntime.

The system shou ld be designed in such a m anner so as minimize the opp ortun ity forleaks of cooling w ater or refrigerants within the comp uter room controlled zone.Piping should not be run in the ceiling void of the room. It is also extremelyimportan t that air cond itioner p iping is not run directly in front of the airconditioner d ischarges. Multiple taps shou ld be d esigned into chilled w ater pip ingto facilitate changes in configurations.

The mechanical supp ort systems should be connected to the building m onitoring

system. Status an d critical alarms shou ld be recorded . This system shou ld alsomonitor the in-room air cond itioners and conditions throu ghout the subject areas.

3.7 Air Distribution Tile PlacementPerforated floor tiles are the sup ply d iffusers of a dow nw ard flow air conditioningsystem. The modular design of the tiles allows for the flexibility necessary in acomplex data center environment. The nu mber a nd type of tiles for each p articularsite will depend on the specific characteristics of the site. The following guidelinesshould, how ever, be considered in th e determ ination.

s Consult your air conditioner m anu facturer or maintenan ce comp any. Most


39/106


y p yman ufacturers recommend a specific amount of opening in th e floor sur face, inthe form of air distribution tiles, for each model they produce. This is based onthe d esired air velocity and the air volum e of the air conditioners. The hard wa re

in the room w ill determine the p recise num ber of tiles needed.s Maintain ap prop riate subfloor p ressure levels. The subfloor p ressure d ifferential

enables efficient conditioned air distribution. It may be necessary to adjust thenu mber of perforated tiles to achieve approp riate pressure levels. Some hard waredesigns, such as some air cooled mainframes or closed cabinet surefire orcommu nications racks, dr aw conditioned air d irectly from the su bfloor void. Thetile cut-outs for these units of hardw are mu st be taken into account w hendeterm ining the nu mber of perforated tiles for air d istribution. If there is a large

amou nt of hard wa re relying on direct subfloor cooling, it may be necessary toredu ce the num ber of air distribution tiles to achieve adequ ate subfloorpressurization.

s Avoid unnecessary holes in the access floor surface. Oversized cable cut-outs orunnecessary holes in the access floor surface will detract from the subfloorpressurization and overall conditioning efficiency. These should be filled willapp ropriately rated air dam pening bags or other ap propr iate fillers.

s Avoid cooling short cycles. Cooling short cycles occur when perforated tiles areplaced between air conditioners and the nearest units of heat-prod ucinghard ware. When this hap pens, cold subfloor air return s to the air cond itionerswithou t ad dressing conditions at th e hard wa re. Because the regulating sensors forthe air cond itioner are located in the retu rn airflow, the air conditioner w illregister that conditions in th e room are cooler than is actually the case, and th eun it may cycle out of its cooling mod e wh ile cond itions in the ha rd ware areas stillcall for conditioning. This affects both temperature and relative humidity.


40/106


FIGURE 3-2 Cooling Short Cycle Air Pattern s

s Form hard ware row s perp endicular to air conditioners. After the conditioned airmixes with the h eat load of the room, it should h ave an un obstructed path ba ck tothe air conditioner return . If the hardw are rows are generally perpend icular to theair conditioners, the air can travel dow n the aisles to the return s. If the rows areparallel to the air conditioners, the return air mu st travel over the hardw are un itsin a less efficient manner. Returns through the open ceiling void of the room

should be avoided. These can affect the efficient response of the air conditionersto the in-room conditions, and expose the controlled areas to the ceiling voidconditions. In add ition th e large volume of the ceiling void m ay have to be takeninto account in certain types of fire suppression systems that rely on preciseconcentrations of gasses to function properly.

s Avoid su bfloor obstructions that could hind er conditioned air distribution Thesubfloor void should not be used for storage of any kind. Cabling should beeffectively managed to avoid jumbled cables or congested areas. Cables piled

from the subfloor deck surface to the bottom of the raised floor can partitionportions of the subfloor, isolating them from the air conditioners.


41/106


FIGURE 3-3 Congested Cabling

3.8 Hardware PlacementThe following recommendations are based on the basic front-to-back air-flowcabinets used by most Sun Microsystems equipment. The installation documentationfor the hard ware being installed should be consulted prior to installation. Thespecific hardware in place in the various subject areas of the facility must be takeninto consideration w hen assessing the environment. The density and complexity ofthe heat load can offer many challenging problems. The necessity of maintainingapp ropriate cond itions in a mainframe environment are u niversally und erstood, but

environmental conditions in server areas are often overlooked. The heat load ofsmall, individu al servers or storage arrays is generally low, but w hen th ese units arestacked in cabinets, the density of the heat load increases dramatically. In addition,the new er technologies utilize den ser geometries in their internal comp onents thanmany older ones. In many instances, large server rooms can have a much higher heatload than traditional mainframe rooms.

The philosophy u sed w hen ad dressing the needs of a single server or storage array

cabinet ma y no longer app ly when large num bers of these cabinets are groupedtogether. Simply m ultiplying the needs of the one to d etermine the n eeds of the

grou p w ill often overlook impor tant factors. The interactions mu st also be taken intoH h i l d i l i h h i ifi


42/106


account. H ow th e un its are placed in relation to one a nother can h ave a significantimpact on h eatload concentration, conditioned air d istribution and serviceaccessibility. While a single unit may be very tolerant, as its needs are more easily

met, a large group requires a great deal of thought and planning to p roperlyaccomm odate, and continu ed vigilance to properly m aintain.

Many stud ies have been condu cted regard ing the changing trends in technology.There is no question that the same compu ting pow er can now be achieved withequipm ent taking u p a fraction of the floor space previously needed . Thu s, acomparison of compu ting or storage capacity versus h eat load w ill indicate asignificant decrease in heat load in recent years since far less equipment is neededfor the same data capacity. What this comparison does not indicate, however, is thatwh ile you now n eed less equipm ent to do the same things, the heat load p er squarefoot is mu ch den ser. Thus, if you fill the sam e amou nt of floor space w ith newequipm ent, you will have much m ore comp uting p ower, but you will also often havea mu ch higher or more concentrated h eat load.

The majority of Sun r acks are designed to take in conditioned sup ply air in the front,pass it over the h eat-loads of the internal compon ents, and exhau st it from th e rear.These racks house a w ide variety of components with differing d esigns. Some

components u tilize airflow p atterns that go from bottom to top , some go from frontto back, and some go from one side to the other. In each of these instances, thecabinet converts the airflow pattern to a front-to-back or front-to-top design. Thisallows for ease of planning in th at it is not necessary to accomm odate m ultipleairflow d esigns. The cabinet p rovides a generally un iform design.

Because of the front-to-back air-flow p atterns, the ideal p lacement wou ld h ave theunits installed face-to-face (and back-to-back) as illustrated in FIGURE 3-4. Byconfiguring the units in th is fashion, direct tran sfer of the h ot exhaust of one u nit

into the intake of another is eliminated . If the aisle wid th is maintained at theappropriate width of 4 feet (1.22 meters), in this scenario, the side-to-side spacingbetween r acks can be red uced virtu ally to zero. Enough clearance should bemaintained to allow a ny rack within that lineup to be removed for service orreplacement. This is possible because the side panels of the Sun cabinets are solid,and not u sed for air flow. If all air cond itioning considerations are app ropriatelyadd ressed, the aisles between th e exhausts can be m ade as narrow as 3 ft. (0.9 m). Ifthe u nits are placed closely together, it is recomm ended that n o more than five (5)

cabinets be grou ped before a m inimu m of 1 1/ 2 feet (.46 m) is left for access to theirrear, or between aisles.


43/106


FIGURE 3-4 Preferred Hardware Placement

If, for some reason, the units must be installed all facing in the same direction (notrecomm ended ), it is imp ortant that adequ ate space be left between the aisles toavoid th e direct tran sfer of the heated air from the exhaust of one un it into the intakeof another as illustrated in FIGURE 3-5. This can be accomplished by allowingadequ ate space between th e units and the aisles. The am ount of space necessary willbe depend ent on nu merous factors in the room. Assum ing that there is adequa te airconditioning , and that it is being efficiently d istributed , 4 feet (1.22 meter s) should

be enough space for the aisles. In some instances, where conditioned air distributionis less than optimal, wider aisles may be necessary. If the width of the aisles is lessthan this recomm endation, it is advised that th ere be a m inimum of 1 ft. (0.3 m)between the units to help spread out the heat load.


44/106


FIGURE 3-5 Alternate H ardw are Placement

It is important to remem ber that the man y aspects of the comp uter roomenvironment are interconnected. Changes to one aspect will, in turn , affect others.Along the same lines, imp lementing one change in isolation may prod uceun desirable results. The overall effects on th e environment of the room mu st betaken into consideration. In either scenario, the following considerations mu st betaken into account:

s It is important that at least one floor tile is removable in each aisle. Though theremay be some flexibility, depending on the configuration, aisles should measureapproximately 1.2 m (4 ft). This width may be slightly higher or lower dependingon tile and cut-out placement.

s Adequ ate spacing m ust be left either between the units or between group s ofunits to allow movem ent behind th e cabinets and between aisles.

s Air distribution tiles should be placed in such a fashion so as to deliverconditioned air to the intake of each cabinet.

s The subfloor pressure d ifferential must be ad equate to d eliver the air to th e areasin need of cond itioning. The subfloor p ressure d ifferential should be m aintainedat an optimal level of 0.05 in wg, though 0.02 in wg is normally acceptable as aminimum level.

s The air conditioning return shou ld be placed in such a fashion that the air has aclear path to return d own the aisle. The air mu st not be forced to travel over thecabinets to get to the retur n.

s Optima l temperatures in the above floor hardw are space should be kept betw een70 to 74 F (21 C to 23 C) and relative hu mid ity levels should be m aintained


45/106


70 to 74 F (21 C to 23 C), and relative hu mid ity levels should be m aintainedbetween 45% and 50% RH. While the hard ware can op erate within a m uch w iderrange, these recomm endations p rovided the greatest buffer against inapp ropriate

influences.It should be noted tha t the preceding recomm enda tions are for hard wa re installed inSun Microsystems cabinets. Componen ts installed outside of these cabinets mayhave very different requirements. Comp onents w ith a side-to-side air-flow p atternare converted to front-to-back air flow by the Sun Cabinets. If these components areinstalled ou tside of these cabinets, it is imp ortan t that th ey not be placed closely nextto each other, as the exhaust of one u nit w ill enter the intake of the next, leading tovery high tem peratu res dow n th e line. For these same reason, it is extremely

importan t that the u nits are not installed in after mar ket or custom cabinets withoutfirst consulting an d gaining app roval from your Sun Microsystems rep resentative.

3.9 Subfloor Pressure Differential

RecommendationsIt is not enough for app ropriate return air temp erature to be maintained by the airconditioning system, the conditioned air must also be efficiently transferred to theareas of the room in need of conditioning. If the conditioned air provided by theenvironmental control equipm ent is not delivered to th e hard wa re properly, the airconditioning system is not p erform ing efficiently. Because the su bfloor void acts as adu ct, with th e air distribution tiles and cable cut-outs acting to d istribute th e

conditioned air, an ad equate subfloor pressure is necessary in order to d eliver thecooled air. If the subfloor pressure is too low, the conditioned air will not be forcedto the areas of the room in need of cooling. Air conditioner m anu facturers n ormallyrecomm end the ap prop riate free area in the form of air d istribution grilles andperforated floor pa nels.

Ideally, the positive pressu rization of the su bfloor void in r elation to the abov e-floorhard wa re space should be maintained at or n e

Documents

Datacenter Planning Guide