HPCフォーラム2015 B-1RandD 100 Award 受賞記念講演常温水冷スパコンHP Apollo 8000開発エンジニアによる誕生秘話 Nicolas Dube Ph.D

© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

HPC フォーラム 2015テクニカルコンピューティング最前線～ HP-CAST Japan ～

R&D 100 Award 受賞記念講演常温水冷スパコンHP Apollo 8000、開発エンジニアによる誕生秘話2015年 4月 24日ヒューレット・パッカードカンパニー最上級テクノロジストNicolas Dobé, Ph.D

© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.2

Proportion of renewable energy grew from 1.12% to 2.3% worldwide from 1990 to 2010.

Fact.

=> Therefore the world is more environmentally friendly

?


Worldwide Energy Situation

Not quite.

• The Bad Side of Ratios:

• Renewables % growing

• But overall fossil fuels absolute dependency is still increasing


IT Energy Consumption

Report to Congress on Server and Data Center Energy Efficiency, August 2007

USA 2013: 10560*1000*24*365 = 92 billion kWh

DataCenter Dynamics Census 2013

Just the beginning of a BIG problem

2013


How big is that?

38 840 MW...

Three Gorges Dam:22 500 MW


Worldwide Datacenter Power: 38 840 MW• 78 Million people, about the population of Germany (@ 2kW per person on average)

• 8.5 Million houses (US average consumption)

• 31 Million electric cars @ 100 km / day

How much energy is that?

In the US alone: 10 560 MW

Source: Make IT Green, Cloud Computing and it’s contribution to climate change, GreenPeace, March 30, 2010

• 39% US Energy is coal

• Coal generates ~1 kg CO2 eq / kWh( “clean coal” is ~ .85 kg CO2 eq / kWh)• 4 118 000 kW * 24 * 365 = 36 billion kWh

• Or 36 Million tons of CO2 emissions / year

• Equivalent carbon footprint of 9 Million people (using world average at 4 tons per capita)


Assess Situation


You can’t optimize what you can’t measurePower Usage Effectiveness (PUE): the first widely accepted datacenter metric (2006)

PUE = Total Datacenter Power

IT Systems Power

Total Datacenter Power

IT Systems Power


The “holy grail” of datacenters operators?

Power Usage Effectiveness

0

1

2

LightingNetworkingPower DistributionCoolingComputer

PUE = Total Datacenter Power

IT Systems Power


Design Datacenter to Minimize Cooling Energy


Liquid moves energy more efficiently.

Datacenter Face off

0.58 bhpFan

0.05 bhpPump

4 gpm


DataCenter Cooling SystemsChillers

Evaporative Towers

Dry Coolers

0.5 – 1.2 kW / Ton

0.05 – 0.1 kW / Ton

+ make up water0.05 – 0.1 kW / Ton

Outlet Water Temp:

Cooling Energy cost:

typically 7°C-12°C

3-5°C of “wet-bulb” temperature

3-5°C of “dry-bulb” temperature


Tokyo Climate“Climate year” psychrometric

chart


Singapore Climate


Stockholm Climate


TokyoAir cooled datacenter


TokyoApollo 8000



for 1MW system in Tokyo

Energy Savings vs Competition

% liquid cooled

Water inlet temp

Compute Energy*

Cooling Energy

Cooling PUE

Air 0% 12ºC4547h

8.76 GWh 2.54 GWh-

1.29

RDHX 100% 15ºC3758h

>8.76 GWh 2.36 GWh7%

1.27

Apollo 8K 100% 30ºC4h (0h @ 31ºC)

<8.76 GWh 0.30 GWh88%

1.03


Energy Savings detailed (Tokyo)1 MW Computing System - Energy savings vs air-cooledAir cooled system: 2 540 425 kWhApollo 8000 system: 305 643 kWhAnnual energy savings: 2 234 782 kWh @ 15 cents = 335 217$ 1 676 086$ savings over 5 years

Heat re-use: 4609 hours / year 4.6 GWh per 1MW of compute capability 691 350$ savings / year @ 15 cents / kWh 3 456 750$ savings over 5 years

Combined: 5 132 836$ Energy savings (1MW, 5 years operation)


Peregrine / “Project Apollo”

The Birth of Apollo 8000

Building a warm water-cooled supercomputing platform


NREL ESIF Concept

• Reveals not just boxes with blinking lights, but the inner workings of the building as well.

• Tour views into pump room and mechanical spaces• Color code pipes, LCD monitors

Datacenter equivalent of the “visible man”


Key Specifications

NREL ESIF Datacenter

• Warm water cooling: (24C)• Water much better working fluid

than air - pumps trump fans.• Utilize high quality waste heat,

(35C+)• +90% IT heat load to liquid.

• High power distribution• 480VAC, Eliminate conversions.

• Think outside the box• Don’t be satisfied with an energy

efficient data center nestled on campus surrounded by inefficient laboratory and office buildings.

• Innovate, integrate, optimize.

Dashboards report instantaneous, seasonal and cumulative PUE/ERE values and multiple other metrics


One wild ride

From concept to production in less than 2 yearsOctober 2011: First “concept” rackJanuary 2012: NREL issues RFP for HVAC + warm-water cooled HPC platformMarch 2012: HP submits RFP response to NRELJuly 2012: NREL awards contract to HPNovember 2012: Initial delivery: compute, storage, switchesFeb 19th 2013: Delivery of prototype system (4 IT racks, 2 CDUs)Feb 26th 2013: 200TF acceptance test passedAugust 2013: Delivery of final Peregrine system (11 IT racks, 6 CDUs)September 11th 2013: Energy Secretary Ernest Moniz dedication eventSeptember 15th 2013: Peregrine goes into production


NREL Prototype


Doing it the “usual” way

Plumbing – January 2013


Facilities plumbing: round 2


Redefining the deployment model

Plumbing – August 2013


Improved modularity and serviceability

Cooling Distribution Unit


NREL PeregrineIn production since September 2013


Next generation ultra-efficient HPC system

The first HPC data center dedicated solely to advancing energy systems integration, renewable energy research, and energy efficiency technologies

New ultra-energy-efficient, petascale HPC system• $1 million in annual energy savings and cost

avoidance through efficiency improvements

• Petascale (one million billion calculations/ second)

• Average PUE of 1.06 or better

• Source of heat for ESIF’s 185,000 square feet of office and lab spaces, as well as the walkways

• 1MW of data center power in under 1,000 sq. ft. => very energy-dense configuration

Designed to support NREL’s mission, address research challenges, reduce risks and accelerate the transformation of our energy system.

In production at the DoE National Renewable Energy Laboratory (NREL)


HP Apollo 8000


Advancing the science of supercomputingThe New HP Apollo 8000 System

Leading teraflops per rack for accelerated results• 4X teraflops/sq. ft. than air-cooled systems• > 250 teraflops/rack

Efficient liquid cooling without the risk• 40% more FLOPS/watt and 28% less

energy than air-cooled systems• Dry-disconnect servers, intelligent Cooling

Distribution Unit (iCDU) monitoring and isolation

Redefining data center energy recycling• Save up to 3,800 tons of CO2/year (790

cars)• Recycle water to heat facility

Scientific Computing• Research computing• Climate modeling• Protein analysis

Manufacturing• Product modeling • Simulations• Material analysis

4X teraflops/sq. ft. 40% more FLOPS/watt

3,800 tons of CO2


Leading performance density

Dry disconnect server trays

HP Apollo 8000 Cooling Circuit

HP Apollo f8000 Rack

HP ProLiant XL730f

2x2P Servers

HP ProLiant XL740f2P+ 2

Accelerators

HP ProLiant XL750f

2P+2 GPUs

Efficient liquid cooling without the risk HP Apollo 8000 iCDU

Rack

HP Apollo 8000 System

HP InfiniBand Switch for Apollo 8000 System


HP ProLiant XL730f Server (2X2P)

Front Node

(2) Intel Xeon E5-2600 Processors per node

Rear Node

(1) Optional SSD per node

(1) IB FDR Adaptor Kit per node

(16) DDR4 DIMM Slots per node

(1) SUV port per node


Reducing the risk of liquid cooling

• Water stays isolated from server nodes and heat transfer is enabled via thermal bus bars

• No “make or break” water connections during node service events

• Integrated controls maintain server components under optimal temperature specifications

Nine Dry Disconnect Thermal Bus Bars assembled into a

“Water Wall”

Dry Disconnect Innovation


Warm air exits the

nodes

Air to Water Integrated Heat

Exchanger

Back

Front

Cool air is generated as if

flows through the HEX. Cold air

enters the front of the trays

Hybrid Cooling Concept – Air SideTop down view of the rack cut-away


HP Apollo 8000 System Technologies

Dry-disconnect servers• 100% water cooled

components• Designed for serviceability

Open door view of 4 f8000, redundant iCDU racks and underfloor plumbing kit

Management infrastructure• HP iLO4, IPMI 2.0 and DCMI 1.0• Rack-level Advanced Power

Manager

Power infrastructure• Up to 80kW per rack• Four 30A 3-phase 380-

480VAC

Intelligent Cooling Distribution Unit• 320 KW power capacity• Integrated controls with active-active failover

Warm water• Closed secondary loop in

CDU• Isolated and open facility

loop

Raised Floor


From loading dock to Linpack in 13 days!

Cyfronet “Prometheus” system installation

12 Compute Racks, 3 CDUs, 1728 servers


Next step: Integrate with the Environment


EnergyReuse LoopSnow melting Building

Heating

Cooling Tower or Dry Cooler

Warm Water Cooled Servers

Chiller less Data Centers


Energy based scheduling and accounting

R&D: Energy-aware scheduling

• Real-time energy monitoring infrastructure

• Resource allocation based on forecasted kWh and heat generation profile

• Accounting and “billing” post-execution• Energy based• IT + datacenter• BTU credits for “hot” water generation in winter


Apollo 8000 - Most Innovative Product of 2014

US Department of Energy 2014 Sustainability Award


Thank you

[email protected]

Technology

HPCフォーラム2015 B-1RandD 100 Award 受賞記念講演 常温水冷スパコンHP Apollo 8000開発エンジニアによる誕生秘話 Nicolas Dube Ph.D

HPCフォーラム2015 B-1RandD 100 Award 受賞記念講演常温水冷スパコンHP Apollo 8000開発エンジニアによる誕生秘話 Nicolas Dube Ph.D