Upload
jiscs-green-ict-programme
View
654
Download
4
Tags:
Embed Size (px)
DESCRIPTION
Summary Background Information Technology Systems Environmental Conditions Air Management Cooling Systems Electrical Systems Other Opportunities for Energy Efficient Design Data Center Metrics & Benchmarking
Citation preview
NREL is a national laboratory of the U.S. Department of Energy, Office of Energy Efficiency and Renewable Energy, operated by the Alliance for Sustainable Energy, LLC.
US Trends in Data Centre Design with NREL Examples of Large Energy Savings
Understanding and Minimising The Costs of Data Centre Based IT Services Conference
University of Liverpool O?o Van Geet, PE June 17, 2013
2
0"
5"
10"
15"
20"
25"
30"
35"
40"
1.00"
1.03"
1.06"
1.09"
1.12"
1.15"
1.18"
1.21"
1.24"
1.27"
1.30"
1.33"
1.36"
1.39"
1.42"
1.45"
1.48"
1.51"
1.54"
1.57"
1.60"
1.63"
1.66"
1.69"
1.72"
1.75"
1.78"
1.81"
1.84"
1.87"
1.90"
1.93"
Cost%in%M
illions%of%D
ollars%
P.U.E.%
Total%Annual%Electrical%Cost:%Compute%+%Facility%
2
Assume ~20MW HPC system & $1M per MW year utility cost
Facility
HPC
Cost and Infrastructure Constraints
3
BPG Table of Contents
• Summary • Background • Informa?on Technology
Systems • Environmental CondiLons • Air Management • Cooling Systems • Electrical Systems • Other Opportuni?es for
Energy Efficient Design • Data Center Metrics &
Benchmarking
4
CPUs ~65C (149F)
GPUs ~75C (167F) Memory
~85C (185F)
CPU, GPU & Memory, represent ~75-‐90% of heat load …
s
s
Safe Temperature Limits
5
Data Center equipment’s environmental condiLons should fall within the ranges established by ASHRAE as published in the Thermal Guidelines book.
Environmental CondiLons
ASHRAE Reference: ASHRAE (2008), (2011)
(@ Equipment Intake) Recommended Allowable
Temperature Data Centers ASHRAE
18° – 27°C
15° – 32°C (A1) 5° – 45°C (A4)
Humidity (RH) Data Centers ASHRAE
5.5°C DP – 60% RH and 15oC DP
20% – 80% RH
Environmental SpecificaLons (°C)
6
2011 ASHRAE Allowable Ranges
Dry Bulb Temperature
7
Psychrometric Bin Analysis
0
0.002
0.004
0.006
0.008
0.01
0.012
0.014
0.016
0.018
0.02
0.022
0.024
0.026
0.028
0.03
0.032
0.034
0.036
30 40 50 60 70 80 90 100 110 120
Hum
idity
Rat
io (l
b W
ater
/lb D
ry A
ir)
Dry-Bulb Temperature (ºF)
Boulder, Colorado TMY3 Weather Data
TMY3 Weather Data
Class 1 Recommended Range
Class 1 Allowable Range
60ºF
50ºF
40ºF
Rel
ativ
e H
umid
ity
80% 60% 40% 20%
100%
70ºF
60ºF
50ºF
40ºF
Rel
ativ
e H
umid
ity
80% 60% 40% 20%
100%
80ºF
Design Condi?ons (0.4%): 91.2 db, 60.6 wb
8
EsLmated Savings
Baseline System DX Cooling with no economizer
Load 1 ton of cooling, constant year-‐round
Efficiency (COP) 3
Total Energy (kWh/yr) 10,270
RECOMMENDED RANGE ALLOWABLE RANGE
Results Hours Energy (kWh) Hours Energy (kWh)
Zone1: DX Cooling Only 25 8 2 1
Zone2: Mul?stage Indirect Evap. + DX (H80) 26 16 4 3
Zone3: Mul?stage Indirect Evap. Only 3 1 0 0
Zone4: Evap. Cooler Only 867 97 510 57
Zone5: Evap. Cooler + Outside Air 6055 417 1656 99
Zone6: Outside Air Only 994 0 4079 0
Zone7: 100% Outside Air 790 0 2509 0
Total 8,760 538 8,760 160
Es0mated % Savings -‐ 95% -‐ 98%
9
Data Center Efficiency Metric
• Power Usage EffecLveness (P.U.E.) is an industry standard data center efficiency metric.
• The raLo of power used or lost by data center facility infrastructure (pumps, lights, fans, conversions, UPS…) to power used by compute.
• Not perfect, some folks play games with it. • 2011 survey esLmates industry average is 1.8. • Typical data center, half of power goes to things other than
compute capability.
9
“IT power” + “Facility power” P.U.E. =
“IT power”
10
PUE – Simple and EffecLve
11
-20
0
20
40
60
80
100
0.75
0.85
0.95
1.05
1.15
1.25
1.35
1.45
Out
door
Tem
pera
ture
(°F)
PUE
Data Center PUE
-20
0
20
40
60
80
100
0.75
0.85
0.95
1.05
1.15
1.25
1.35
1.45
Out
door
Tem
pera
ture
(°F)
PUE
Data Center PUE
Outdoor Temperature
“I am re-‐using waste heat from my data center on another part of my site and my PUE is 0.8!”
ASHRAE & friends (DOE, EPA, TGG, 7x24, etc..) do not allow reused energy in PUE & PUE is
always >1.0.
Another metric has been developed by The Green Grid +; ERE – Energy Reuse EffecLveness.
h?p://www.thegreengrid.org/en/Global/Content/white-‐papers/ERE
13
ERE – Adds Energy Reuse
Utility
Cooling
UPS PDU
IT
RejectedEnergy
(a)
(b)(c) (d)
(f)
(e)
ReusedEnergy
(g)
14
Credit: Haselden ConstrucLon
• More than 1300 people in DOE office space on NREL’s campus
• 33,445 m2
• Design/build process with required energy goals
50% energy savings from code LEED Pla?num
• Replicable Process Technologies Cost
• Site, source, carbon, cost ZEB:B Includes plugs loads and datacenter
• Firm fixed price -‐ US $22.8/m2 construcLon cost (not including $2.5/m2 for PV from PPA/ARRA)
• Opened June 10, 2010 (First Phase)
DOE/NREL Research Support Facility
15
RSF Datacenter
• Fully containing hot aisle – Custom aisle floor and door seals – Ensure equipment designed for
cold aisle containment § And installed to pull cold air
Ø Not hot air… – 1.18 annual PUE – ERE = 0.9
• Control hot aisle based on return temperature of ~90F.
• Waste heat used to heat building. • Outside air and EvaporaLve
Cooling • Low fan energy design • 176 Sq m.
Credit: Marjorie Scho?/NREL
16 16
17 17
Data Center Load GROWTH (40+ kW in 2 years) since NO recharge!
18
Move to Liquid Cooling
• Server fans are inefficient and noisy. – Liquid doors are an improvement but we can do
beger!
• Power densiLes are rising making component-‐level liquid cooling soluLons more appropriate.
• Liquid Benefit – Thermal stability, reduced component failures. – Beger waste heat re-‐use op?ons. – Warm water cooling, reduce/eliminate
condensa?on. – Provide cooling with higher temperature coolant.
• Eliminate expensive & inefficient chillers. • Save wasted fan energy and use it for
compuLng. • Unlock your cores and overclock to increase
throughput!
19
Liquid Cooling – Overview
Water and other liquids (dielectrics, glycols and refrigerants) may be used for heat removal. • Liquids typically use LESS transport energy (14.36 Air to Water Horsepower ra?o for example below).
• Liquid-‐to-‐liquid heat exchangers have closer approach temps than Liquid-‐to-‐air (coils), yielding increased outside air hours.
20
2011 ASHRAE Liquid Cooling Guidelines
NREL ESIF HPC (HP hardware) using 24 C supply, 40 C return –W4/W5
21
NREL HPC Data Center
Showcase Facility • 10MW, 929 m2 • Leverage favorable climate • Use direct water to rack
cooling • DC manager responsible for
ALL DC cost including energy! • Waste heat captured and used
to heat labs & offices. • World’s most energy efficient
data center, PUE 1.06! • Lower CapEx and OpEx.
Leveraged exper0se in energy efficient buildings to focus on showcase data center.
Chips to bricks approach
• Opera?onal 1-‐2013, Petascale+ HPC Capability in 8-‐2013
• 20-‐year planning horizon 5 to 6 HPC genera?ons.
High Performance CompuLng
22
CriLcal Data Center Specs
• Warm water cooling, 24C Water much beger working fluid than air -‐ pumps trump fans.
U?lize high quality waste heat, 40C or warmer.
+90% IT heat load to liquid.
• High power distribuLon 480VAC, Eliminate conversions.
• Think outside the box Don’t be sa?sfied with an energy efficient data center nestled on campus surrounded by inefficient laboratory and office buildings.
Innovate, integrate, op?mize.
Dashboards report instantaneous, seasonal and cumulaLve PUE values.
23
• Data center equivalent of the “visible man” – Reveal not just boxes with blinky lights, but the inner workings of
the building as well. – Tour views into pump room and mechanical spaces – Color code pipes, LCD monitors
NREL ESIF Data Center Cross SecLon
24
• 2.5 MW – Day one capacity (Utility $500K/yr/MW)
• 10 MW – Ultimate Capacity
• Petaflop • No Vapor Compression
for Cooling
Data Center
25
Summer Cooling Mode
PUE – Typical Data Center = 1.5 – 2.0 NREL ESIF= 1.04 * 30% more energy efficient than your typical “green” data center
Data Center
26
Winter Cooling Mode
ERE – Energy Reuse Effectiveness How efficient are we using the waste heat to heat the rest of the building? NREL ESIF= .7 (we use 30% of waste heat) (more with future campus loops)
Future Campus Heating Loop
Future Campus Heating Loop
High Bay Heating Loop
Office Heating Loop
Conference Heating Loop
Data Center
27
95 deg Air
75 deg Air
• Water to rack Cooling for High Performance Computers handles 90% of total load
• Air Cooling for Legacy Equipment handles 10% of total Load
Data Center – Cooling Strategy
28
PUE 1.0X -‐-‐ Focus on the “1”
Facility PUE
IT Power Consumption
Energy Re-use
We all know how to do this!
True efficiency requires 3-D optimization.
29
Facility PUE
IT Power Consumption
Energy Re-use
We all know how to do this!
Increased work per watt Reduce or eliminate fans Component level heat exchange Newest processors are more efficient.
True efficiency requires 3-D optimization.
PUE 1.0X -‐-‐ Focus on the “1”
30
Facility PUE
IT Power Consumption
Energy Re-use True efficiency requires 3-D optimization.
We all know how to do this!
Increased work per watt Reduce or eliminate fans Component level heat exchange Newest processors are more efficient.
Direct liquid cooling, Higher return water temps Holistic view of data center planning
PUE 1.0X -‐-‐ Focus on the “1”
31
What’s Next?
ü Energy Efficient supporLng infrastructure. ü Pumps, large pipes, high voltage (380 to 480) electrical to rack
ü Efficient HPC for planned workload. ü Capture and re-‐use waste heat.
Can we manage and “opLmize” workflows, with varied job mix, within a given energy “budget”? Can we do this as part of a larger “ecosystem”?
31 Steve Hammond
32
Other Factors
32 5
DemandSMART: Comprehensive Demand Response Balancing supply and demand on the electricity grid is difficult and expensive. End users that provide a balancing resource are compensated for the service.
Annual Electricity Demand As a Percent of Available Capacity
50%
100%
Winter Spring Summer Fall
75%
25%
90%
4MW solar
Use waste heat
Better rates, shed load
DC as part of Campus Energy System
33
ParLng Thoughts
• Energy Efficient Data Centers – been there, done that – We know how, let’s just apply best practices. – Don’t fear H20: Liquid cooling will be increasingly prevalent.
• Metrics will lead us into sustainability – If you don’t measure/monitor it, you can’t manage it. – As PUE has done; ERE, Carbon Use Effectiveness (CUE), etc. will help drive
sustainability. • Energy Efficient and Sustainable Computing – it’s all about the “1”
– 1.0 or 0.06? Where do we focus? Compute & Energy Reuse. • Holistic approaches to Energy Management.
– Lots of open research questions. – Projects may get an energy allocation rather than a node-hour allocation.
34
Otto VanGeet 303.384.7369
NREL RSF 50% of code energy use Net zero annual energy $22.8/m2 ConstrucLon Cost
QUESTIONS?
35
• Thermoelectric power generaLon (coal, oil, natural gas and nuclear) consumes about 1.1 gallon per kW hour, on average.
• This amounts to about 9.6 M gallons per MW year.
• We esLmate about 2.5 M gallons water consumed per MW year for on-‐site evaporaLve cooling towers at NREL.
• If chillers need 0.2MW per MW of HPC power, then chillers have an impact of 2.375M gallons per year per MW.
• Actuals will depend on your site, but evap. cooling doesn’t necessarily result in a net increase in water use.
• Low Energy use = Lower water use. Energy Reuse uses NO water!
Water ConsideraLons
“We shouldn’t use evaporative cooling, water is scarce.”
NREL PIX 00181
36
Data Center Efficiency
• Choices regarding power, packaging, cooling, and energy recovery in data centers drive TCO.
• Why should we care? • Carbon footprint. • Water usage. • Mega$ per MW year. • Cost: OpEx ~ IT CapEx!
• A less efficient data center takes away power and dollars that could otherwise be used for compute capability.
37
HolisLc Thinking • Approach to Cooling: Air vs Liquid and where?
– Components, Liquid Doors or CRACs, …
• What is your “ambient” Temperature? – 55F, 65F, 75F, 85F, 95F, 105F … – 13C, 18C, 24C, 30C, 35C, 40.5C …
• Electrical distribuLon: – 208v or 480v?
• “Waste” Heat: – How hot? Liquid or Air? Throw it away or Use it?
38
Liquid Cooling – New ConsideraLons
• Air Cooling – Humidity – Fan failures – Air side economizers, par?culates
• Liquid Cooling – pH & bacteria – Dissolved solids – Corrosion inhibitors, etc.
• When considering liquid cooled systems, insist that providers adhere to the latest ASHRAE water quality spec or it could be costly.
39
2011 ASHRAE Liquid Cooling Guidelines
40
2011 ASHRAE Thermal Guidelines
2011 Thermal Guidelines for Data Processing Environments – Expanded Data Center Classes and Usage Guidance. White paper prepared by ASHRAE Technical Commi?ee TC 9.9
41
Energy Savings PotenLal: Economizer Cooling
Energy savings poten?al for recommended envelope, Stage 1: Economizer Cooling.12
(Source: Billy Roberts, NREL)
42
Data Center Energy
• Data centers are energy intensive faciliLes. – 10-‐100x more energy intensive than an office.
– Server racks well in excess of 30kW.
– Power and cooling constraints in exis?ng facili?es.
• Data Center inefficiency steals power that would otherwise support compute capability.
• Important to have DC manager responsible for ALL DC cost including energy!
43
Energy Savings PotenLal: Economizer + Direct EvaporaLve Cooling
Energy savings poten?al for recommended envelope, Stage 2: Economizer + Direct Evap. Cooling.12
(Source: Billy Roberts, NREL)
44
Energy Savings PotenLal: Economizer + Direct Evap. + MulLstage Indirect Evap. Cooling
Energy savings poten?al for recommended envelope, Stage 3: Economizer + Direct
Evap. + Mul?stage Indirect Evap. Cooling.12 (Source: Billy Roberts, NREL)
45
Data Center Energy Efficiency
• ASHRAE 90.1 2011 requires economizer in most data centers. • ASHRAE Standard 90.4P, Energy Standard for Data Centers and
Telecommunica0ons Buildings • PURPOSE: To establish the minimum energy efficiency requirements
of Data Centers and TelecommunicaLons Buildings, for: • Design, construcLon, and a plan for operaLon and maintenance • SCOPE: This Standard applies to: • New, new addiLons, and modificaLons to Data Centers and
TelecommunicaLons Buildings or porLons thereof and their systems • Will set minimum PUE based on climate. • More detail at : h?ps://www.ashrae.org/news/2013/ashrae-‐seeks-‐
input-‐on-‐revisions-‐to-‐data-‐centers-‐in-‐90-‐1-‐energy-‐standard-‐scope
46
1. Reduce the IT load -‐ VirtualizaLon & ConsolidaLon (up to 80% reducLon) 2. Implement contained hot aisle and cold aisle layout.
Curtains, equipment configura?on, blank panels, cable entrance/exit ports,
3. Install economizer (air or water) and evaporaLve cooling (direct or indirect). 4. Raise discharge air temperature. Install VFD’s on all computer room air
condiLoning (CRAC) fans (if used) and network the controls. 5. Reuse data center waste heat if possible. 6. Raise the chilled water (if used) set-‐point.
Increasing chiller water temp by 1°C reduces chiller energy use by about 3%
7. Install high efficiency equipment including UPS, power supplies, etc.. 8. Move chilled water as close to server as possible (direct liquid cooling). 9. Consider centralized high efficiency water cooled chiller plant
Air-‐cooled = 2.9 COP, water-‐cooled = 7.8 COP
Energy ConservaLon Measures
47
Equipment Environmental SpecificaLon
Air Inlet to IT Equipment is the important specification to meet
Outlet temperature is not important to IT Equipment
48
Recommended Range (Statement of Reliability) Preferred facility opera?on; most values should be within this range.
Allowable Range (Statement of FuncLonality) Robustness of equipment; no values should be outside this range.
MAX ALLOWABLE
RACK INTAKE TEMPERATURE
MAX RECOMMENDED Over-‐Temp
Recommended Range
Under-‐Temp
MIN RECOMMENDED
MIN ALLOWABLE
Allowable Range
Key Nomenclature
49
Improve Air Management
• Typically, more air circulated than required.
• Air mixing and short circuiLng leads to: – Low supply temperature – Low Delta T
• Use hot and cold aisles. • Improve isolaLon of hot
and cold aisles. – Reduce fan energy – Improve air-‐condi?oning
efficiency – Increase cooling capacity
49
Hot aisle/cold aisle configuraLon decreases mixing of intake &
exhaust air, promoLng efficiency.
Source: hNp://www1.eere.energy.gov/femp/pdfs/eedatacenterbestprac0ces.pdf
50
Isolate Cold and Hot Aisles
Source: hNp://www1.eere.energy.gov/femp/pdfs/eedatacenterbestprac0ces.pdf
70-80ºF vs. 45-55ºF
95-105ºF vs. 60-70ºF
51
Adding Air Curtains for Hot/Cold IsolaLon
Photo used with permission from the NaLonal Snow and Ice Data Center. h?p://www.nrel.gov/docs/fy12osL/53939.pdf
Courtesy of Henry Coles, Lawrence Berkeley NaLonal Laboratory
53
Three (3) Cooling Device Categories
ITEquipment
Rack
cooling water
rackcontainment
SERVER FRONT
1 -‐ Rack Cooler • APC-‐water • Knürr(CoolTherm)-‐water • Knürr(CoolLoop)-‐water • Rigal-‐water
2 -‐ Row Cooler • APC(2*)-‐water • Liebert-‐refrigerant
ITEquipment
Rack
ITEquipment
Rack
ITEquipment
Rack
rowcontainment
cooling water
cooling waterSERVER
FRONT
3 -‐ Passive Door Cooler • IBM-‐water • Vege/Coolcentric-‐water • Liebert-‐refrigerant • SUN-‐refrigerant
SERVER FRONT
ITEquipment
Rackcooling water
Courtesy of Henry Coles, Lawrence Berkeley Na0onal Laboratory
54
“Chill-‐off 2” EvaluaLon of Close-‐coupled Cooling SoluLons
Courtesy of Geoffrey Bell and Henry Coles, Lawrence Berkeley Na0onal Laboratory
less energy use
55
Cooling Takeaways…
• Use a central plant (e.g. chiller/CRAHs) vs. CRAC units • Use centralized controls on CRAC/CRAH units to prevent
simultaneous humidifying and dehumidifying. • Move to liquid cooling (room, row, rack, chip) • Consider VSDs on fans, pumps, chillers, and towers • Use air-‐ or water-‐side free cooling. • Expand humidity range and improve humidity control (or
disconnect).