O 2 Pt2 The O 2 facility will consist of 3 major parts: 1.Read-Out – First-Level Processors (FLP) Connected to the detector frontend Will stay in CR1 (actual computing room located in the LHC access shaft in the SX hall bld 2285) and utilize the racks, power and cooling upgraded/installed during LS1. 2.Event Processing Nodes (EPN), Quality Control (QC), Data Storage (DS) Connected to Read-Out via IT network (fibers) Large installation (~2MW), needs a new computing room Present CRs (CR1 and CR2) were built for LEP/L3, they do not provide an adequate environment for the whole O2 facility: lack of Rack-Space, power, cooling Weight issue (CRs are suspended in the access shaft) 3.Services, Network ALICE O2 3
Future O 2 LHC P2 U. FUCHS, P. VANDE VYVRE ALICE O 2 Project
Data of all interactions shipped from detector to online farm in
triggerless continuous mode Data volume reduction by cluster finder
No event discarded Average factor 2.2(factor 2.5 for the TPC data)
Asynchronous event reconstruction with finalCalibration with a
delay of few hours. HI run 1.1 TByte/s Data Storage: 1 year of
compressed data Bandwidth: Write 90 GB/s Read 90 GB/s Capacity: 60
PB Tier 0 (Computing Centre Meyrin) 20 GByte/s Tiers 1 and Analysis
Facilities Data volume reduction by tracking All the events go to
data storage Average factor 5.5 (factor 8 for the TPC data) 500
GByte/s 90 GByte/s Global computing needs: ~ 100 k CPU cores Use of
accelerators: ~ 5000 GPUs and ~ 500 FPGAs ~ 60 PB of storage FLP
EPN DS ALICE O22 O 2 Pt2 The O 2 facility will consist of 3 major
parts: 1.Read-Out First-Level Processors (FLP) Connected to the
detector frontend Will stay in CR1 (actual computing room located
in the LHC access shaft in the SX hall bld 2285) and utilize the
racks, power and cooling upgraded/installed during LS1. 2.Event
Processing Nodes (EPN), Quality Control (QC), Data Storage (DS)
Connected to Read-Out via IT network (fibers) Large installation
(~2MW), needs a new computing room Present CRs (CR1 and CR2) were
built for LEP/L3, they do not provide an adequate environment for
the whole O2 facility: lack of Rack-Space, power, cooling Weight
issue (CRs are suspended in the access shaft) 3.Services, Network
ALICE O2 3 Computing room CR0 A new computing room (CR0) is needed:
Space Requirement Min U, possible in 45 large racks Power
Requirement IT: 115 kW UPS kW Normal Power Total
(IT+Infrastructure): 240 kW UPS kW Normal Power Cooling Requirement
Min. 2.5 MW cooling capacity Foreseen Placement: Immediate vicinity
of hall 2285 to re-use a maximum of services already present and to
reduce costs for cabling ALICE O2 4 CR0 Tentative Location ALICE O2
5 CR0 Tentative Location 24m 27m ALICE O2 6 Power Needs ALICE O2 7
Power Needs Evolution Present DAQ + HLT Systems ( ) Future O2
System ( ) Servers (box count) 600~2000 Server Max. Power
Consumption W1.1 kW Max. # of servers per rack 2554 Max. power per
rack kW32-50 kW Normal Power (avail/used) [kW] 540 / 0x / 2062 UPS
Power (avail/used) [kW] 500 / / 453 ALICE O2 8 O 2 Pt2, Power Needs
LocationPower UPS CR1 Power Normal CR1 Power UPS CR0 Power Normal
CR0 FLPCR1213 EPNCR01650 QCCR015 DSCR ServicesCR1+CR0515
NetworkCR1+CR01045 IT TOTAL CoolingCR0125 G TOTAL ALICE O2 9 O 2
Pt2, Power Needs The power needs of the O2 System: UPS Power
(ALICE: 500kW available for O 2 ): 228 kW needed in present CR1
(all in place) 225 kW needed outside for CR0 (TBD) Normal Power: No
need for normal power in CR1 2 MW of additional normal power needed
outside for CR0 (TBD) Possibility of adding 500 kW ? (hardware
evolution) The installation will start in ~Q So the power should be
available ~1 year before. ALICE O2 10 Schedule ALICE O2 11 Q Start
of installation in CR0 Computing Room 0 (CR0) Options ALICE O2 12
CR0: building options Option (I): Container-based Foundation
(concrete slab) and services (electricity and water) needed
Container has different noise screening than a building, probably
additional noise screens are needed Containers are fast to deploy
(6-8 months including site preparation) and to extend (4 months)
Option (II): A new building Civil engineering work for a new
building (can be a simple sandwich wall structure) Electricity
distribution in building and to racks (e.g. Canalis) Engineering of
a Cooling System depending on the cooling solution chosen ALICE O2
13 Examples of container computing rooms ALICE O2 14 Cooling
Options ALICE O2 15 Operating temperatures of IT equipment IT
equipment quite efficient at exhausting its own heat Operating
temperature is the inlet temperature IT fans generally bottom out
at around 24 C Dell recommended temperature in 2009: C (1) Most of
todays IT equipment can run at > 30 C inlet temperature, some
even up to 43 C (1) Data Center Operating Temperature: What Does
Dell Recommend, Dell, 2009 (2) Data Center Operating Temperature:
The Sweet Spot, Dell, 2011 (3) 2011 Thermal Guidelines for Data
Processing Environments Expanded Data Center Classes and Usage
Guidance; ASHRAE TC9.9; 2011: ALICE O2 16 CR0: cooling options CR0
shall have a Power Usage Effectiveness (PUE) value of better/lower
than 1.15 The IT power needs are the same for all solutions Several
options are being studied based on cooling technology: I.A
Container based Solution 1.Chilled-Water Cooling 2.Primary-Water
Cooling 3.Free-Air Cooling II.A new Building 1.Rack-Cooling with
Chilled Water 2.Rack-Cooling with Primary Water 3.Free-Air Cooling
ALICE O2 17 O 2 Pt2, Cooling Options Cooling Option (1): Chilled
Water In General Needs a chilled water computing room than can
accept a load of 2.5 MW Running chillers is not a GreenIT practice,
PUE will be [ ] Needs additional power to be installed for chillers
In a building: Racks with water-cooled back door exist up to 32 kW
only e.g. KNURR CoolTherm, 38U usable, 80cm wide, 130cm deep (74/84
usable not deep enough, min 92cm needed) Not very space efficient,
the footprint of the computing room will be big In a Container:
Different Rack/Back Door or In Row cooling options exist Same
constraints (kW/rack, U usable..) as for a building ALICE O2 18 O 2
Pt2, Cooling Options Cooling Option (2): Primary Water (24 C, max
28 C) In General Primary water can be used directly to cool
equipment ? If not: additional heat exchanger needed (+3 C) Direct
Liquid Cooling (DLC) excluded due to type of equipment (no solution
for main heat contributors: disks, GPUs) In a building: No Rack
cooling solution found for racks>32kW using 22/24/27 C water In
a Container: A solution exists for water at ~ 22 C, so additional
chillers would be needed to cool down from primary water
temperature Better than chilled water but still not GreenIT -
friendly ALICE O2 19 O 2 Pt2, Cooling Options Cooling Option (3):
Free Air Cooling (FAC) In General State-of-the-art solution, used
by many large computing centers (EBay, Google (1), Microsoft, YAHOO
(2),..) The O2 power density is probably higher (1) Googles
Chiller-Less Data Center, Data Center Knowledge, July 15, 2009 (2)
Yahoo Computing Coop: Shape of Things to Come? Data Center
Knowledge, April 26, In a building: Needs a special building
geometry to optimize the airflow. Hot-aisle separation and
immediate heat evacuation (roof) Challenge: Temperature/Humidity
control ALICE O2 20 O 2 Pt2, Cooling Options Cooling Option (3):
Free Air Cooling (FAC) In a Container: Two major suppliers
identified: DELL and HP following different philosophies: DELL:
outside-airflow with recycling to control temperature and humidity,
chillers can be added but are not an integral part HP: closed air
volume for IT cooled by chillers running in FreeCooling ALICE O2 21
O 2 Pt2, Cooling Options Cooling Option (3): Free Air Cooling (FAC)
In a Container, contd PUE value depends on IT installed in
container DELL solution can achieve HP runs at 1.15 year-average
(due to chillers) Additional Cooling is available (if outside too
hot) DELL: evaporative cooling (default), DX chillers if needed HP:
air-air chillers Racks are vendor-neutral vis--vis installed IT
equipment Power density of 50 kW/rack achievable ALICE O2 22 Free
cooling map Geneva Free air in Geneva:
