Site Personnel
Site Director: Dr. Yogendra Joshi
Faculty team:
• Dr. Satish Kumar
• Dr. Minami Yoda
Researchers/Students:
• Vaibhav K. Arghode (Post Doc)
• Jayati Athavale (Ph.D.)
• Yunji Gu (M.S.)
Research Focus Areas
• Air flow imaging • Air delivery from floor tiles – measurements and modeling • Server air flow, temperature, measurements and modeling • Dynamic sensing and controls for energy usage optimization • Reduced order modeling • Liquid cooling • Waste heat recovery • Containerized data centers
Perforated Tile
Air Flow
Rate = 0.234
m3/s (496
CFM)
• A 20kW rack requires ~ 1m3/s (2,118 CFM) with a temperature difference of 20°C across the rack
• Inlet velocities ~O(0.3 (top)-1.2 (tile surface) m/s
Y. Joshi and P. Kumar, Eds., Energy Efficient Thermal Management of Data Centers, Springer, 2012
Particle Image Velocimetry
Perforated Tile Air Flow Rate = 1.224 m3/s (2594 CFM)
Inlet velocities ~O(5.83 (top)-7.5 (tile surface) m/s
Bottom servers up to the height of 500 mm from the floor do not receive cool air
Air entrainment velocity in the cold aisle (shown white dotted) has increased from ~ 0.6m/s (case 3) to 1.8m/s (case 4) severely disrupting the air distribution to the opposite rack
Increased severity of reversed flow in the servers located in the bottom of the rack (v~1.5 to 2.5 m/s). Reversed flow height increases to ~ 400 mm
Pressure outlet
Pressure inlet (K=10)
Tile
Grill
Fan
Mass flow inlet Symmetry
Gap
Pressure inlet
Pressure inlet
Tile
Aisle top
Ais
le
Experiments (PIV)
Geometrical Resolution (GR)
Velocity (m/s) Modified Body
Force (MBF)
Porous Jump (PJ)
Tile
Top (-0.8%)
Rac
k
Tile
Top (-46.2%)
Tile
Top (-43.5%)
CFD/Reduced Order Model of Tiles Data Center Lab
6
2500 CFM of air flow through both Tile and Rack
• MBF model able to capture prominent flow features from PIV and GR model
[ref] Arghode, V. K., Joshi, Y., Room Level Modeling of Air Flow in a Contained Data Center Aisle, ASME Journal of Electronic Packaging, v 136, p 011011-1-10, 2014.
Air Delivery from Tile to Adjacent Rack (PIV)
Tile Flow = Rack Flow
Umag/Uin
7
Data Center Lab
K = P/(0.5Uin2)
Ais
le
Tile (K=9.7)
Top (67.7%)
36.7% 1/4”
1,177 CFM
Tile (K=38.3)
Top (100.1%)
Porosity 21.1%
Tile (K=9.2)
Top (64.6%)
Pore size
1/8”
• Lower porosity higher air by-pass • Smaller pore size non-negligible effect on flow field • Higher flow rate minimal effect
Rac
k
Tile (K=9.7)
Top (72.5%)
Flow rate 1,766 CFM
[ref] Arghode, V. K., Joshi, Y., Experimental Investigation of Air Flow through Perforated Tile in a Raised Floor Data Center, (ISTP 2013) International Symposium on Transport Phenomena, 01-05 November 2013, Yamaguchi, Japan.
Cold Aisle Containment Studies 8
Closed Aisle
Temperature (C)
Open Aisle
1,14 2,13 3,12 4,11 5,10 6,9 7,8
EXP, closed, 1 only
1,14 2,13 3,12 4,11 5,10 6,9 7,8
EXP, open, 1 only
79% 89%
50%
75%
100%
125%
150%
Open, 1 Only Closed, 1 Only
Flo
w R
ate
(%)
Total Tile/Rack Air FLow Rate
• Containing cold aisle improves the thermal
field uniformity
[ref] Arghode, V. K., Sundaralingam, V., Joshi, Y., Phelps, W., Thermal Characteristics of Open and Contained Data Center Cold Aisle, Journal of Heat Transfer, v 135, p 061901-1-11, 2013.
T8 T9 T10 T11 T12 T13 T14
T1 T2 T3 T4 T5 T6 T7
V8 V9 V10 V11 V12 V13 V14
V1 V2 V3 V4 V5 V6 V7
V21 V20 V19 V18 V17 V16 V15
Rack8
Rack9
Rack10
Rack11
Rack12
Rack13
Rack7
Rack6
Rack5
Rack4
Rack3
Rack2
Rack14
R
ack1
CRAC
1 - ON
CRAC
3 - OFF
CRAC
2 - OFF
Storage
Networking
Empty
1U
Data Center Lab
CEETHERM Data Center Layout
Power
distribution
unit
Power
distribution
unit
D
O
O
R
Storage
640 node
rack
Storage
IBM Blade
Center
IBM Blade
Center
IBM Blade
Center
IBM Blade
Center
Dell Power
edge 64 Node
Rack
Network
IBM Blade
Center
IBM Blade
Center
IBM Blade
Center
IBM Blade
Center
IBM Blade
Center
IBM Blade
Center
Spare
35
71
3
24
14
HP rack
HP rack
IBM rack
Mixed HP/
IBM rack
Mixed HP/
IBM rack
HP rack
IBM Blade
Center rack
IBM rack
D
O
O
R
Server
Simulator
6
91
1
8
IBM Blade
Center rack
10
12
Downflow
CRAC
Upflow
CRAC
Upflow
CRAC
Downflow
CRAC
Downflow CRACDownflow CRAC
1
Power
distribution
unit
Experimental zone Computational zonea
8.7
5m
6.4m 6.4m
APC In row
cooler
APC In row
cooler
Equipment Donations courtesy:
Racks: APC
Fan assist Chimney Racks: Wrightline
Servers: Intel and OIT
CRAC unit: Liebert
In row coolers: APC (Critical Components)
Server Simulator: APC
Branch circuit power metering: PDI corp
Remote KVM switch: Minicom and Digi
IBM Blade Center Details:
• 10 Racks
• 6 Servers / Rack
• 14 Blades / Server
• 4 Cores/ Blade
• Total: 3360 Cores
Data Center Laboratory Layout
•3-D Stereoscopic PIV (Particle Image Velocimetry system for room level air flow mapping.
•25kW Server Simulator with adjustable fan and heater settings to simulate a variety of heat loads.
•Perforated tiles with adjustable dampers to control air discharge rate.
Experimental Facilities
3-D PIV system
Server Simulator
Fan speed and Heat
setting dials
Perforated floor tiles with dampers
•One of the first PIV systems for rack level air flow mapping
•Customized to scan the desired plane and stitch vector maps together to get full
flow field
Particle Image Velocimetry System (PIV)
PIV system
Perforated floor tile
Measurement plane area
(0.61x1.98) m2
Camera 1 (Image map A)
Camera 2 (Image map B)
Light Sheet
Laser Traverse
Aisle Center
Aisle Top
Rack Inlet
X
Y
Sever Rack
Light
Sheet
∆ t
Cross correlation function
Flow seeded
with
particles
Pulsed Laser
(Nd:YAG)
Cylindrical lens
Image map 1
pulse 1
Image map 2
pulse 2
∆x, ∆y ( Particle Displacement)
Noise filtering and
random vector
elimination
Vector
Statistics
Data Analysis
Velocity
Stream lines
Vorticity
Turbulence
CCD camera
Vector map
Interrogation
area
Δt
ΔxU
Δt
ΔyV
Velocity vectors
Data Center Lab
Laser Cameras
Traverse
Rack
Tile
Aisle
Top
2 ft
6 ft 3”
1 2
Grid based Temperature Measurement
Cold Aisle (252 thermocouples) Hot Aisle (126 thermocouples)
• T type thermocouples with wire gauge 32
• Covers width of the cold/hot aisle and height of the server racks
13
Data Center Lab
Other Measurement Tools
• Tile Air Flow Rate Measurement
• Measurement of dynamic pressure head to obtain velocity, compensation for flow resistance through the hood
• Rack Air Flow Rate Measurement (Custom, Degree C)
• Measurement of velocity using thermal anemometer
Thermistor + Anemometer (15 × 3)
0.0
0.2
0.4
0.6
0.8
1.0
1.2
0.0 0.2 0.4 0.6 0.8 1.0 1.2
Cal
ori
me
tric
Flo
w R
ate
(kg/
s)
Anemometic Flow Rate (kg/s)
Server SimulatorIBM Blade Center
±10%
14
Data Center Lab
• Plenum Pressure Measurement (Degree C) • Mass flow measured based on differential
temperature over an heated surface & correlated to differential pressure
• Compensation for connecting tube lengths
Cold Aisle Containment
CRAC-
2-D
CRAC-
3-U
CRAC-
1-D
Return
Return
PDU-2
PDU-1
Racks
Racks
Fully Contained Aisle Open Aisle
Door at aisle
entrance
Deployable
curtain at the top
Door at aisle
entrance
Blanking
panel for
front
Network
rack
Racks with
rear door
heat
exchangers
• Physically separating the hot and cold aisle
can result in uniform server inlet temperature
15
Data Center Lab
• Perforated tiles used to supply air from plenum to the room space
• Parameters porosity, pore size and shape, anterior dampers or fins
Different Tile Designs Under Investigation
“Generic” “Slotted” “Chamfered” “Fan”
TOP
BOTTOM
Data Center Lab
Dynamic Events in Data Centers
17
• Fluctuating IT load
760
780
800
820
0 5 10 15 20 25 30
Po
we
r (W
)
Time (min)
VM Power Profile
Courtesy Junwei Li, CERCS, GT
Liu et al., Phil.
Trans. R. Soc.
A 2012 370
Microsoft Live Messenger
Dynamic Resource Allocation
18
Armbrust et al., 2009, Report UCB/EECS-
2009-28
Loss of cooling resources ( Lower CRAC set
points than required)
Over-Provisioning