Using CFD Analysis to Predict Cooling System Performance in …mcrowley/tmp/apc_cfd_webcast.pdf · 2008. 4. 12. · What is CFD? CFD stands for Computational Fluid Dynamics Fluid

© 2006 American Power Conversion Corporation. All rights reserved. All trademarks provided are the property of their respective owners.

Using CFD Analysis to Predict Cooling System Performance in Data Centers

Ben Steinberg, P.E.Staff Applications Engineer


Agenda

What is CFD?Applications of CFDApplicability to the Data CenterHow it worksExamples


What is CFD?

CFD stands for Computational Fluid DynamicsFluid Dynamics is the study of the motion of fluids (liquids and gases)Complex equations that describe fluid flow and heat transfer were developed hundreds of years agoFor simple systems, these equations can be solved manually


Applications of CFDComputers make solving these equations possible for very complex systems.Developed in the 1960’s for aerospace, defense and nuclear power industries.Usage grew into the automotive industry in 1970’s.CFD is used as a “virtual wind tunnel” for designing and testing airplanes, cars and ships before a prototype is even built.Even used by hospitals to simulate flows of fluids through the human body.


Applicability to the Data CenterIn 1965, Intel co-founder Gordon Moore predicted that the number of transistors on a chip would double about every 2 years.A musical birthday card costing a few U.S. dollars today has more computing power than the fastest mainframes of a few decades ago.A single rack of servers can now consume as much power and create as much heat as 3 household ovens!


Increasing Heat Densities

05

101520253035

'95 '96 '97 '98 '99 '00 '01 '02 '03 '04 '05

Year

Per R

ack P

ower

Den

sity (

KW)

Max per-rack IT Load

Data Center rated rack power capacity

IT Loads are greatly exceeding rated capacity!


Survey Question #1

Do you currently have hot spots in your (or a customer’s) data center?

YesNo


Applicability to the Data Center: PastIn general, the heat load of a data center was (and still is) about equal to the power consumption of the IT equipment, in addition to other miscellaneous loads (lighting, solar, etc.)In the past, ensuring that the installed cooling capacity was at least as much as the heat load was enough to cool the IT equipment.Raised floors and overhead ducts were the most common types of air distribution.Perforated tiles in raised floors or supply grilles in overhead ducts were placed as necessary to supply cool air to the equipment inlets.


Applicability to the Data Center: Present (and Future)

Today’s high density loads require so much cooling and airflow (up to 7 tons and 2,400 CFM per rack) that it is difficult to supply the correct quantities of cool air where it is needed (see next slide).Hot spots are created in areas where not enough cool air is supplied.Reactions to hot spots include lowering CRAC set points and adding more CRAC units, both of which can make things worse.Every data center is unique:- Size, shape, type of equipment- Changes very frequentlyVery difficult to predict performance and level of redundancy.


Grate tilePerf tile

Blade Ser

vers

Standard IT Eq

uipment

WithEffort

TypicalCapability Extreme Impractical

12

10

8

6

4

2

00 100 200 300 400 500 600 700 800 900 1000

[47.2] [94.4] [141.6] [188.8] [236.0] [283.2] [330.4] [377.6] [424.8] [471.9]

Floor Tile Cooling Ability Requires careful raised floor design, careful CRAC placement,

and control of under-floor obstacles (pipes/wiring)300-500 cfm

RackPower(kW)

that can becooled by one tile with this

airflow


How it worksApplicable to existing as well as planned installations.All equipment is entered into a 3D computer model- Physical size- Power and cooling capacity- Air flowWhen the model is complete, the computer solves the equations.- could take hours or days, depending on size and detail.After solving, results can be manipulated to show temperatures, pressures and flows in different locations.Model should be maintained as equipment changes take place.


CFD Examples


Raised floor cooling CFD exampleDesign Parameters:

4kW per rack / 40 racks total Hot Aisle/Cold Aisle layoutTotal power 160kWRoom size: 38' x 38' x 14' high2' raised floor w/25% perforated tiles in front of racks and PDUs5 CRAC units N+1 redundancy at room level - 4 units running at the same time = 160kWCRAC units are standard chilled water 40kW (12 ton) downflow


Raised floor cooling CFD exampleH

OT

AIR

HO

T AI

R

COLD AIR/PERFORATED TILES




Raised floor cooling CFD example

Redundancy:

Any unit failure results in loss of

cooling to an area

N+1 at room level does provide

adequate cooling in failure modes

Sectional Plane @ 5’-6” from Raised Floor


Survey Question #2

What is your (or your customer’s) highest rack density today?

Less than 1 kW per rack1 - 2 kW per rack3 - 4 kW per rack5 - 6 kW per rackMore than 6 kW per rackNot sure


Hard floor cooling CFD exampleDesign Parameters:4kW per rack / 40 racks total Total power 160kWRoom size: 38' x 38' x 14' highRaised floor not used for air distribution6 in-row cooling unitsN+1 redundancy at row level - 4 units running at the same time = 160kW


InRow Architecture

Hard floor cooling CFD example

CO

LD A

IR

HO

T AI

R

CO

LD A

IR

HO

T AI

R

CO

LD A

IR

Net

wor

kAir

IR

Net

wor

kAir

IRN

etw

orkA

ir IR

Net

wor

kAir

IR

Net

wor

kAir

IRN

etw

orkA

ir IR


Sectional Plane @ 5’-6” from Datacenter Floor

InRow Architecture

Hard floor cooling CFD example

Redundancy:

Normal Operating

Mode 4 units running

No Hot Spots

developed in front of

IT equipment.


Raised floor cooling CFD exampleDesign Parameters:

2kW per rackNot Hot Aisle/Cold Aisle layoutTotal power 138kWRoom size: 53' x 41' x 9' high2' raised floor w/25% perforated tiles in front of racks4 CRAC units N+1 redundancy at room level - 3 units running at the same time = 150kWCRAC units are glycol cooled DX 50kW (15 ton) downflow


Temperature DistributionTop View, top half of racks


Temperature DistributionSide View

•Cool air from floor plenum is consumed by bottom half of racks; top half is left to ingest hot air from recirculation.


Temperature DistributionTop View, top half of racks

•New CRAC unit is added; note the cooler temperatures around the room.


Under floor pressure & flow CFD example

1 foot raised floor height2kW per rackAnalysis of 2 different CRAC layouts


Temperatures

Scenario 1 Scenario 2


Plenum Airflow

Raised floor hidden for clarity.



Plenum Pressure

Raised floor and IT racks hidden for clarity.Note dead spots at center of recirculations.



Perforated Tile Flow Rates

flow rate per tile (cfm)200 425 650 875 1100

speed (ft/min)•Very Non-uniform distribution

•BACKFLOW here



Conclusions

As equipment power densities rise, so does the amount of heat produced.Very difficult to accurately predict performance and redundancy at high densities.Recent advances in computer power and software have made CFD a viable tool to help fix or prevent hot spots from occurring in the data center.


Using CFD Analysis to Predict Cooling System Performance in Data Centers

QUESTIONS


Using CFD Analysis to Predict Cooling System Performance in Data CentersAgendaWhat is CFD?Applications of CFDApplicability to the Data CenterIncreasing Heat DensitiesSurvey Question #1Applicability to the Data Center: PastApplicability to the Data Center: Present (and Future)How it worksCFD ExamplesRaised floor cooling CFD exampleSurvey Question #2Hard floor cooling CFD exampleRaised floor cooling CFD exampleTemperature DistributionTop View, top half of racksTemperature DistributionSide ViewTemperature DistributionTop View, top half of racksUnder floor pressure & flow CFD exampleConclusions

Documents

Using CFD Analysis to Predict Cooling System Performance in …mcrowley/tmp/apc_cfd_webcast.pdf · 2008. 4. 12. · What is CFD? CFD stands for Computational Fluid Dynamics Fluid