Embed Size (px)
Citation preview
Data Center Trends and
Power Management
Roger SchmidtIBM Chief Thermal Architect
Sept 12, 2006
Contents
• Power Trends/Impacts on Data Center• IBM’s Product Innovation• Industry Actions• Final thoughts
What Customers are Asking
• I am out of power in my data center, so what can be done?
• How do I handle the hot spots in my data center resulting from high server powers?
• How do I get real time information from servers on temperature and power so I can run my data center more energy efficiently?
Facility/Network Concerns
0% 5% 10% 15% 20% 25% 30% 35%
None of the above
Poor location
Excessive facility cost
Insufficient raised floor
Insufficient power
Excessive heat
“Power and cooling will be a top 3 issue with all CIO’s in the
next 6-12 months”
Michael Bell – Gartner Group
Everyone Has A “Computer Room”
Branch Office Client/Web Hosting Site
IT Equipment is HOT…And HEAVY
P – Series 595
31”W x 66”D x 80”H
23 kW
3014 Lbs Toyota Camry
3276 Lbs
P-Series 690
62” W x 50” D x 80”H
32 kW
4233 Lbs
New Datacom Equipment Power Density Chart
Rack Level Heat Load TrendRack Level Heat Loads Rising at an Exponential Rate
Year of Announcement1994 1996 1998 2000 2002 2004 2006
5
10
15
20
25
30
35R
ack
Leve
l Hea
t Loa
d (k
W)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
z900
z990
p595
p575
p520
BladeProcessor / CacheMemoryIO SubsystemCoolingPower
System Power ComparisonSystem Power Varies as a Function of Configuration and Workload
It is NOT Just the Processor
Basic Data Center Air Flow
A/Cunit
CONCRETE SUBFLOOR
RAISED FLOOR AREA
Server Rack
PEDESTAL
COLD INPUTAIR
HOT OUTPUT RETURN AIR
1
43
2
- Perforated tiles do not support high flowrates required by high density servers - Underfloor obstructions from chilled water pipes and cables
Air Blockages beneath Raised Floor
Chilled Water PipesCables
Chilled air does not reach top of racks
ColdAisleColdAisle
ColdAisle
ColdAisle Hot
Aisle
The Problem in Data Centers
CFD Model of a Raised Floor Data Center
Highly complex flow patterns and localized rack inlet air heatingfocus of many studies by R. Schmidt and Co-workers (1997-2004)
Computational Fluid Dynamics
$0
$10
$20
$30
$40
$50
$60
$70
$80
$90
1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
Installed base(M units)
0
2
4
6
8
10
12
14
16
18
U.S. MarketSpending(US$B)
New server spendingPower and cooling
The New Economics of IT; A Paradigm Shift
• Power and cooling spend will exceed new server spending (Gartner 2006)
2000 – Raw processing Raw processing ““horsepowerhorsepower”” is the primary is the primary goal, while the goal, while the infrastructure to support it infrastructure to support it is assumed readyis assumed ready
2006 –– Raw processing Raw processing ““horsepowerhorsepower”” is a given, is a given, but the infrastructure to but the infrastructure to support deployment is a support deployment is a limiting factorlimiting factor
Three Cooling Challenges 1. The System2. The Rack3. The Datacenter
Contents
• Power Trends/Impacts on Data Center• IBM’s Product Innovation• Industry Actions• Final thoughts
IBM delivers business value with innovation at all levels
Infrastructure Solutions
Grid Computing Supercomputing Capacity on Demand
SOAInformation on Demand
Capacity on DemandInfrastructure
IBM z/Architecture™
Power Architecture™
Cell Broadband Engine™ X-Architecture™
x86
Storage BladesMainframe
Technology & Packaging
ProductFamilies
•Track Record of Consistent Predictable Delivery
IBM POWER Architecture
POWER4+
POWER4
POWER5
POWER5+
POWER6
20012003
20042006
2007
Power 6 will provide leadership for years to come…
Innovations in all areas of system design
System Energy Management
RAS
Performance
Flexibility
Features and function
Technology
Current development Status
Increased performance with increased energy efficiency
The Datacenter
Hot Spot Area
Cool blue
Solution
Rear Door Heat Exchanger
•• IBMs focus on power efficiency allows IBMs focus on power efficiency allows our systems to run with less power our systems to run with less power than our competitorsthan our competitors– For some customers this level of efficiency is
still inadequate• Older data center floors were not
designed to handle the heat load that today's servers place on it
• Many customers run out of power and cooling before running out of rack space
For these customers, IBM has developed a suite of additional tools to employ;
–– IBM Cool Blue rack based heat IBM Cool Blue rack based heat eXchangereXchanger
–– ““Smart ToolsSmart Tools””• Power Calculator, Power Executive • Virtualization
Understanding Where Costs are Critical
•• IBMIBM’’s Cool Blue Rear Door s Cool Blue Rear Door Heat eXchanger can remove Heat eXchanger can remove over 50% of a rackover 50% of a rack’’s heat s heat outputoutput
– No new fans or electricity needed.
– Attaches to back of rack (adds 5”)
– No rearrangement of datacenter
– Cost effective; 1KW cooling = $286
•• The Cool Blue Heat The Cool Blue Heat eXchanger adds cooling eXchanger adds cooling capacity at ~1/4 the cost of capacity at ~1/4 the cost of traditional methodstraditional methods
Rear Door Heat Exchange
Front Cold
BackHot
Cable Opening
Subfloor
Underfloor Chilled Air
Air flow
Perf tile Tile floor
water lines
Rear Door Heat Xchanger
IBM Enterprise Rack
Front Cold
BackHot
Cable Opening
Subfloor Underfloor Chilled Air
Air flow
Perf tile Tile floor
Normal Operation of Server
Improved Operation of Server
Comparison – with and without Cool Blue
Typical Data Center Power Distributionand Energy Savings with Water
Other15.1%
Lighting2.7%
Office Space Conditionin1.5%
Electrical Room Cooling4.4%
Cooling Tower Plant4.3%
Data Center Cooling 15.7%
Server Load56.3%
Other12.4%
Lighting2.2%
Office Space Conditioning1.3%
Electrical Room Cooling3.6%
Cooling Tower Plant3.5%
Data Center Cooling 30.8%
Server Load46.2%
Air Cooled Server Racks
Water Cooled Server Racks
Description
ElectricityConsumption(kW)
with Air CooledServers
ElectricityConsumption(kW)with Water Cooled
ServersOther 402 402Lighting 73 73Office Space Conditio 41 41Electrical Room Coo 118 118Cooling Tower Plant 114 114Data Center Cooling 1000 418Server Load 1500 1500Total Load 3248 2666
Example:
Savings of ½ million dollars/year at $0.10 / kw-hr
Livermore Data Center – 3rd Fastest Supercomputer
8585.8
85.1
8685.2
87.2
86.1
86.5
80.7
83.1
80.1
77.9
77.8
77.9
76.7
75
High Power Rack Clusters within 10,000 sq ft can require 10 MW
including supporting power and cooling
Annual Energy Cost for 10 MW at $0.10/kW-hr = $8.8 M
New Building Cost for 10,000 sq ft = $15 M
AK 10.99
CA 11.53
TX 7.90
HI 16.19
MN 6.31
NY 12.98
NC 6.70
GERMANY7.7
UK6.7
FRANCE5.0
ITALY16.2
BRAZIL 4.7
ARGENTINA3.3
CHINA15.0
JAPAN12.7
KOREA5.9
AUSTRALIA6.1
NEW ZEALAND6.0
TAIWAN5.7
Energy Cost Around the World(US cents/kw-hr)
•Deep Global Expertise•Dedicated team since 1986•Mechanical & Electrical Engineers, Space planners and PM’s on staff
• Most with over 20 years experience• A team of 450 people globally
•Designed and built over 30 million SF of raise floor from Boulder and Brussels to Bangalore
•We have learned what works … because we use it•Built and run over 400 of our own datacenters, with 6M sq ft currently in operation
•Built over 120 of our own disaster recovery facilities that support our clients
•Built and reside in 4 of top 10 green buildings in Japan•Built state of the art clean rooms for our own leading processortechnology manufacturing
•Clear understanding of current technologies and a vision of future with tight linkage to product development and research teams
•Well established partnerships with key providers ensuring best choices to fill your requirements including: APC, Liebert, etc.
IBM Site and Facilities Services- Providing Data Center Solutions
IBM and the Energy Efficient Data Center
• IBM servers provide leadership energy efficiency todayPower5 efficiency continuing to Power6Server power management features - PowerExecutive
• IBM is delivering leadership energy management technologyEnable customers to maximize datacenter computational efficiencyEnable customers to minimize datacenter operating costs
• IBM has the skill and resources to assist our customers in planning and deploying data center solutions
IBM’s long term commitment to providing solutions for our customers
Source:Brad McCredie, Analyst meeting, July 20, 2006
Contents
• Power Trends/Impacts on Data Center• IBM’s Product Innovation• Industry Actions• Final thoughts
Worldwide Focus on Climate/Energy• Energy Usage becoming a critical global issue
– Climate Change– Energy Security and Availability– Energy Cost
• Attracting the attention of regulators, legislators, and customers• Carbon caps• Energy Efficiency
– Energy Saving Law – Japan– Energy Star– EPA researching server farm energy usage
• Green grid – Best practices in a data center
• Benchmarks on power/performance
IBM Providing Industry Leadership- ASHRAE TC9.9 Committee
Current practices in Datacom Facility TCO analysis and Energy Efficiency
High density data centers - case studies and other considerationsContamination in datacom equipment center
Seismic considerations in datacom equipment centers
Other Books being developed
Chair of committee : Roger SchmidtIBM Chief Thermal Architect
Source: Roger Schmidt
Thermal Guidelines Book –Overview
People from these firms participated: ANCIS, APC, Ceyba, Cisco, CRS, Cray, DataAire, Dell, DLB, Echelon, EMC, Engineered Refrigeration Sys, Flomercs Inc, Fujitsu, Hellmer Medved Engineers, Hitachi, HP, IBM, Innovative Research, Intel, LBNL Labs, Liebert, Lucent, Mallory & Evans, Motorola, NCR, Sun, Telecordia, Unisys, Uptime Institute
Chapter 1 – Introduction
Chapter 2 – Equipment Environment Specifications
Chapter 3 –Environmental Measurement
Chapter 4 – Equipment Placement & Airflow Patterns
Chapter 5 – Equipment Manufacturer’s Heat & Airflow Reporting
Power Trend Book - OverviewChapter 1 – Introduction
Chapter 2 – Background
Chapter 3 – Load Trends & their Applications
Chapter 4 – Air Cooling of Computer Equipment
Chapter 5 – Liquid Cooling of Computer Equipment
Appendix A – Collection of Terms
Appendix B – Additional Trend Chart Information / Data
Appendix C – Electronics, Semiconductors, Microprocessors, ITRS
Appendix D – Micro Macro Overview of Datacom Equipment Packaging
©2005 ASHRAE TC9.9
People from these firms participated: Alcatel, ANCIS, ATI, Cisco, Cray, DataAire, Dell, DOD, DLB, EMC, EYP, Fanniemae, Freescale, Fujitsu, HellmerMedved Engineers, HP, IBM, Intel, LBNL Labs, Liebert, Mallory & Evans, Motorola, Nortel, Sun, Syska & Hennesey, Taylor Engineering, Uptime Institute
Data Center Considerations - OverviewPart 1 Datacom Facility Basics
Chapter 1 Introduction; Chapter 2 Design Criteria; Chapter 3 HVAC Load Considerations; Chapter 4 Computer Room Cooling Overview; Chapter 5 Air Distribution; Chapter 6 Liquid Cooling
Part 2 Other Considerations
Chapter 7 Ancillary Spaces; Chapter 8 Contamination; Chapter 9 Acoustical Noise Emissions; Chapter 10 Structural & Seismic; Chapter 11 Fire Detection & Suppression; Chapter 12 Commissioning; Chapter 13 Availability & Redundancy; Chapter 14 Energy Efficiency
ANCIS, APC, Bell South, Citigroup, Data Aire, Dell, DOD, DLB, EDS, EYP, Fannie Mae, Fluent, Fujitsu, Heapy Engineering, HP, IBM, Intel, LBNL, Liebert, Mallory & Evans, Nelson Acoustical, Nortel, Rice University, Stulz, Sun, Syska & Hennessy, Tier 4 Consulting, Wright Line
People from these firms participated
Liquid Cooling Book – Overview
People from these firms participated:APC, Aavid, Cray, DataAire, Dell, DLB, EYP, Hellmer Medved Engineers, HP, IBM, Intel, Liebert, Lytron, Mallory and Evans, NCR, NSA, Panduit, Rittal, Sanmina, SGI, Spraycool, Sun, Trane
Chapter 1 – Introduction
Chapter 2 – Cooling Services for Equipment Cooling Systems
Chapter 3 – Facility Piping Architecture
Chapter 4 – Liquid Cooling Implementation for Datacom Equipment
Chapter 5 – Liquid Cooling Infrastructure Requirements for Chilled Water Systems
Chapter 6 – Liquid Cooling Infrastructure Requirements for Chilled Water Systems and Datacom Equipment Cooling Systems
Contents
• Power Trends/Impacts on Data Center• IBM’s Product Innovation• Industry Actions• Final thoughts
Final Thoughts•• Thermal issues will become ever more Thermal issues will become ever more
severe over time, and fundamental severe over time, and fundamental innovations are required to provide a route innovations are required to provide a route forwardforward
•• Power efficient products as revealed today Power efficient products as revealed today resolve customer challenges already limiting resolve customer challenges already limiting their ability to leverage existing assetstheir ability to leverage existing assets
•• This is not going to get any easierThis is not going to get any easier
