Brainserve Datacenter: the High-Density Choice

BrainServe Datacenter: the high-density choice

Swiss Big Data User Group Meeting – 24.06.2013

Agenda

BrainServe in very brief

The high-density choice: back in 2008

Cooling technologies: low-density and high-density

Water-based cooling

BrainServe example

High-density and energy efficiency

Q&A


BrainServe in very brief

Swiss based independent datacenter in the northwestern part of Lausanne

Physical housing space for IT equipment

Collaboration with IT providers and telecom operators

Dedicated plot owned by BrainServe

Location with a low risk profile

Datacenter design and building « from scratch » (2008 – 2010)

Highly secure and resilient environment (2’000 m2 of housing areas) 1’000 m2 of «classical» air cooling

1’000 m2 of high-density liquid cooling


The high-density choice: back in year 2008

Emergence of data-centric universe Already more data, and still increasing

Storage, processing needs and access requirements are increasing accordingly

Hardware evolution: power densification Historically (1970 – 2005): smaller transistors used less electricity to operate

exponentially better performance & density for a constant power envelope (“free energy”)

Since 2005: continue to make transistors smaller, but they use similar electricity to operate

chip energy consumption is shooting up (end of “free energy”)

Electrical input = heat output increase in computing power = increase in energy consumption = increase in heat dissipation

It is often not possible nor economical to spread the load across the space increase in computing power = increase in heat density requires high-density cooling


Cooling technologies: low-density


Traditionally: air is the primary heat transport media

Heat exchanger are away from the IT racks

Air is blown into the plenum under the raised floor and comes out in front of the racks (preferably on a cold aisle containment)

In standard configuration (traditional room height): 4 kW/rack on average

Technical and operational constraints may lead to very disparate densities and hot spots that are difficult to handle

Cooling technologies: high-density

High-density applications require typically 10 kW to 20 kW per rack

Different manufacturers and designs for high-density cooling: in-row cooling, in-rack cooling, overhead cooling, rear door cooling

Identical central feature: bring the heat exchanger closer to the server rack Reduced volume of air to be moved

Shorter and more predictable air flows

Handling of hot spots easier

Better use of the cooling capacities

Most of the products require connections to a chilled water loop


Water-based cooling

Why do we use water?

Air is abundant and easy to handle, but not a very good heat conductor

To transfer more heat we need to dramatically increase the volume of air transported

Water is a much better heat conductor


Sourc

e:

André

Opperm

ann

, S

WIN

OG

-26

BrainServe example (1)

1’000 m2 (50% of IT space) equipped for high-density

N+N redundancy (production, distribution and cooling units) Necessary to cope with rise of temperature in case of failure

In-row cooling with hot aisle containment

Designed for 10 kW/rack on average, easily scalable to 50 kW/rack


BrainServe example (2)

Great modularity Positioning of the rack cooling matches the specific needs

Proportions of IT racks and cooling units are tightly coupled

Automatic chilled water network isolation in case of major failure

Water leak detection system

10 MVA electrical feed

Power distribution with 400 A and 630 A busbars

Fire detection and automatic extinction

BrainServe is an industrial partner for the EXTREME (Energy- and thermal-aware design of many-core heterogeneous datacenters) research project with the EPFL that look for future energy efficient design for IT and datacenters


High-density and energy efficiency

Energy efficiency is a major concern for datacenters

Among others, high-density is a factor of efficiency improvement

A datacenter is most efficient when it is fully loaded, and the IT load has the biggest impact on the global efficiency (bigger than e.g. the outside temperature) Densification of the IT load improves the energy efficiency: the more energy we use, the more

energy we save

By bringing the heat exchanger closer to the heat source, we keep the air volume to be moved as low as possible Power consumption is proportional to the cube of the fan speed

By increasing the ΔT on the heat exchanger (e.g. with hot aisle containment), we lower the air volume to be moved for a given load

High-density does not mean lower temperature in the IT room ASHRAE recommended environmental envelope is independent from the power density

High-density means more re-usable heat


Gabriel Boissonnard T +41 21 637 69 04 M [email protected] W www.brainserve.ch

THANK YOU FOR YOUR ATTENTION!


mailto:[email protected]

http://www.brainserve.ch/

Technology

Brainserve Datacenter: the High-Density Choice