ALICE trigger and computing upgrade: design, technology

ALICE trigger and computing upgrade: design, technology and performance

B. von Haller for the ALICE Collaboration

6.10.2016

CERN

Outline

▶ ALICE trigger & computing upgrade in a nutshell▶ Triggers, Heart Beats and Timeframes▶ Hardware and computing model▶ Nodes and network▶ Hardware acceleration▶ O2 Facility and Tiers

▶ Software ▶ Paradigms▶ Development processes and tools

▶ Project organisation▶ Schedule

B. von Haller | ALICE Collaboration | 6.10.2016 1

ALICE Trigger/DAQ Upgrade in a nutshell

Requirements1. After LS2, LHC min bias PbPb at 50 kHz

▶ ~100 x more data than during Run 1

➔ Too much data to be stored

2. Physics topics addressed by ALICE upgrade

▶ Rare processes

▶ Very small signal over background ratio

▶ Needs large statistics of reconstructed events

▶ 13nb-1 for PbPb

➔Triggering techniques very inefficient if not impossible in most cases

3. TPC inherent rate (drift time ~100 µs) < 50 kHz

➔Support for continuous read-out (TPC), as well as triggered read-out


New computing system• Read-out the data of all

interactions• Compress these data

intelligently by reconstructing them online

• One common online-offline computing system: O2

Paradigm shift compared to approach for Run 1 and 2

Functional flow


Asynchronous (few hours)event reconstruction withfinal calibration

Compressed Sub-Timeframes

Continuous and triggered streams of raw data

Data aggregationSynchronous globaldata processing

Data storage (60 PB)1 year of compressed dataWrite 170 GB/s, Read 270 GB/s

Compressed Timeframes

Reconstructed eventsCompressed Timeframes

Base Line correction, zero suppr.ReadoutData aggregation Local data processing

Detectors electronics

3.4 TB/s HI:50kHz pp/pA:200kHz

500 GB/s HI:50kHz pp/pA:200kHz

90 GB/s HI:50kHz pp/pA:200kHz

Tier 0, Tiers 1 and Analysis facilities

20 GB/s

Trigger system

First Level Processors (FLP)

Event Processing Nodes (EPN)

Continuous & triggered read-out

GBT

→data &

←trigger &

←configuration

CommonReadout

Unit (CRU)

PCIe bus in FLP

→data &

←configuration

O2

&DCS

TTC PON

FIT

TPC

ITS

MCH

MID

MFT

TOF

TPC

TRD

ZDC

Trigger system


Central Trigger Processor (CTP)

FIT ZDC ACO TOF EMC PHS

O2.. Online and Offline Computing System

FLP.. First level processor

DCS.. Detector Control System

TTS.. Trigger and Timing Distribution System

CTP.. Central Trigger Processor

GBT.. Gigabit Transceiver

FTL .. Fast Trigger LinksTTC .. Timing Trigger Control

←trigger (ITS/MFT/TRD)

GBT

Triggered read-out

CRORCDDL1 or 2

→data &

←configuration

O2

&DCS

←trigger

PCIe bus in FLP

→data &

←configuration

TTC

ACO

CPV

EMC

HMP

PHS

Trigger system


• Broadcast and continuous serial stream

• LHC Bunch Clock synchronous• Send clock

• Time Division Multiplexing• Shared bandwidth, lower line rate• LHC Bunch Clock synchronous• Send busy/throttling

Optical Network Unit



Optical Line Terminal

+

+

+

+

Run 1 and 2 : • TTC (Timing Trigger Control) used for trigger distribution (unidirectional)• Busy/throttling transported electrically on a separate link

Run 3 : • TTC PON

for detectors with continuous readout

PON: Passive Optical NetworkCommercial system adaptedfor the CERN requirements

S. Baron

Triggers, Heart Beats, Timeframe


CRU

(& frontend) Time

Heart Beat Frames (HBF): data stream delimited by two HBs

HBF and TF rates programmable

Typical values:

- HB: 1 per orbit, 89.4 µs: ~10 kHz

- TF: 1 every ~20 ms: ~50 Hz

- 1 TF = ~256 HBF

EPN Time Frame (TF):

grouping of all STFs from all FLPs for the same time period

from triggered or continuously read out detectors

FLP Sub-Time Frame (STF) in FLP 0:

grouping of (~256) consecutive HBFs from one FLP FLP 1FLP n

Read-out: Continuous

Heart Beat (HB)issued in continuous & triggered

modes to all detectors

Physics triggercan be sent to upgraded

detectors will be sent to non-

upgraded detectors

and triggered

Trigger data fragments

System throttling


CTP

FLPCRU

CTP FLP

ReadoutLinks

HBF/trigger acknowledge/message

3b

HBF/Trigger Transmitted ?

2

HB trigger/message1

HBF3a

HB Map1 1 1 0 1 1 0 1 1 1

HB Map1 1 1 0 1 1 0 1 1 1

Delete HBF from bufferDiscard incoming data

4 Delete HBF from memory

5

HB Map 6

O2 Hardware facility


Detectors

9000 Read-out Links

270 FLPsFirst LevelProcessors

(FLPs)

1500 EPNsEvent Processing

Nodes(EPNs)

Input: 270 portsOutput : 1500 ports

3.4 TB/s

SwitchingNetwork

500 GB/s

RD and WR 440 GB/s

68 StorageArrays

60 PB

34 StorageServers

StorageNetwork

Input: 1500 portsOutput : 34 ports

90 GB/s

Readout & FPGA Hardware acceleration


RORC 1 C-RORC CRU

2 ch @ 2 Gb/sPCIe gen.1 x4 (1 GB/s)

12 ch @ up to 6 Gb/sPCIe gen.2 x 8 (4 GB/s)

24 ch @ 5 Gb/sPCIe gen.3 X 16 (16 GB/s)

Custom DDL protocol Custom DDL protocol(same protocol but faster)

GBT

Protocol handlingTPC Cluster Finder



Common-Mode correctionZero suppression

Run 3LS 2Run 2LS1Run 1

Hardware acceleration (FPGA)


Performance of the FPGA-based FastClusterFinder algorithm for DDL1 (Run1) and DDL2 (Run2) compared to the software implementation on a recent server PC.

Hardware acceleration (GPU)

▶ For TPC track finding on EPNs (as today’s HLT)

▶ Possibly more use cases depending on R&D


0

0.5

1

1.5

2

2.5

3

3.5

4

0.5x10^6 1x10^6 1.5x10^6 2x10^6

Tra

ckin

g T

ime (

s)

Number of Clusters

HLT GPU TrackerHLT CPU Tracker

Tracking time of HLT TPC Cellular Automata tracker on Nehalem CPU (6Cores) and NVIDIA Fermi GPU.

Network performance tests


Comparison of Ethernet, IP over InfiniBand and IP over Omni-Path

40 GbE: 40 Gigabit EthernetOPA: Intel® Omni-PathIB: InfiniBandO2: ALICE Online-Offline framework

0

20

40

60

80

100

Ne

two

rk T

hro

ugh

pu

t (G

b/s

)

Block Size (B)

40 GbE (O2) 40 GbE (max) OPA (O2) OPA (max) IB (O2) IB (max)

1K 10K 100K 1MB 10M 100M

StorageClient File Systems performance tests


CLI

1,10,40 GE / IB

...

40 GE / IB

10 GE

DS

12G SAS

DSMDMDADM

ADM: Administration serverMD: Metadata serverDS: Data serverCLI: ClientGE: Gigabit EthernetIB: InfiniBandGPFS: General Parallel File System Lustre: open-source parallel file system Ceph: distributed object store RADOS: Reliable Autonomic Distributed Object Store

O2 Farm

▶ ~100k CPUs, ~5k GPUs, ~500 FPGAs▶ FLPs at P2 in existing CR1▶ EPNs and storage need a new

dedicated room▶ Space and weight limitations

▶ Two scenarios▶ CR0

▶ Container(s)▶ Call for tender (common with LHCb

and neutrino platform)

▶ Common Data Center in Prevessin▶ An alternative to the CR0 at P2 has

been proposed by CERN▶ New common data center in Prevessin▶ Solution based on the GSI Green Cube▶ Being studied


CR1

CR0

IT network

9000optical fibers

Computing Model


T0/T1

CTF -> ESD -> AOD

AF

AOD -> HISTO

O2

RAW -> CTF -> ESD -> AOD

1

T2

MC -> CTF -> ESD -> AOD

1..n

1..n 1..3

CTF

AOD

AOD

AOD

Glossary• RAW: raw data• CTF: Compressed Time

Frame• ESD: Event Summary Data• AOD: Analysis Data Object• O2: Online-Offline facility• T0, T1, T2: Grid Tier 0, 1, 2• AF: Analysis Facility• MC: Monte-Carlo • HISTO: Subset of AOD

specific for a given analysis

Software

▶ Message-based multi-processing ▶ Ease of development▶ Ease to scale horizontally▶ Possibility to extend with

different hardware▶ Multi-threading possible

within processes

▶ ALFA : ALICE-FAIR concurrency framework ▶ Data transport layer▶ ZeroMQ▶ Multi-process▶ Steady development

▶ AliceO2▶ Prototyping▶ Development started

Design


Libraries and tools

ALFA

Cbm AliceO2Panda

FairRoot

. . . . . . .

EPN m

EPN 1

n

2

1

…

…

Software

▶ Especially important▶ Most software will run online

▶ Large code base

▶ Many people involved, from different backgrounds▶ That is also a blessing as we have a large expertise to pick from

Development processes


MasterFork

Automatic testsGuidelines compliance

Human review

Developer

Pull Request

Production Deployment

O2 Organization

▶ Today

▶ 3 distinct projects : DAQ, HLT, Offline

▶ 2 different code base : HLT+Offline and DAQ

▶ 2 different reconstructions : HLT and Offline

▶ ALICE O2

▶ 1 project

▶ 1 code base

▶ 1 reconstruction


O2 Organization

▶ Steering board▶ Project leaders of the DAQ, HLT and

Offline

▶ 13 Computing working groups (CWGs) ▶ Made of people from the DAQ, HLT and

Offline projects ▶ Plus participants from many detectors

▶ Plenaries▶ Successful transition

▶ Cross-fertilization of the different cultures

▶ There is a strong and growing sense of belonging


O2 Schedule


2020-2021

DesignR&DDemonstrators

O2 TDRProducts selection

Detailed designR&DPrototypingDevelopment

Products selectionPrototypes

Detailed designR&DPrototypingDevelopment

Products selectionPrototypesFinal componentsDeploymentCommissioning

Development

Products selection

Final componentsDeploymentCommissioningProduction

2018-20192016-20172014-20152012-2013

High-Level DesignR&D

Trigger TDRProject organization

Run 3LS 2Run 2LS1Run 1

Summary

▶ ALICE Trigger and computing upgrade is an exciting project with very ambitious requirements▶ 13nb-1 for PbPb

▶ >3TB/s continuous and triggered detector input

▶ New trigger and throttling strategy to handle the continuous readout

▶ Major paradigm change with combined offline and online system

▶ Hardware system▶ HW acceleration (FPGAs, GPUs)

▶ O2 farm with ~100 k CPU cores and ~5000 GPUs

▶ Software development▶ Collaboration with FAIR team in Germany and JPARC experiments in Japan allows a

steady progress for the framework

▶ People from different groups and backgrounds work together▶ We all improved by doing so !

▶ On track despite a challenging schedule



O2/T0/T1 T0/T1

ArchiveCTFAOD

Storage

EPNs O(1000)

FLPsFLPs O(100)

O2 architecture (1)

B. von Haller | O2 Project | 19.05.2015 23

Raw data input

Local processing

Frame dispatch

Global processing

Compressed timeframes

Partially compressedsub-timeframes

Storage

Syn

chro

no

us

Data Reduction 0

e.g. clustering

Sub-timeframes

Calibration 0 on local data,

ie. partial detector

Time slicing

Buffering

Local aggregation QC

Tagging

Detector reconstruction

e.g. track finding

Timeframe building

Full timeframe

Data Reduction 1

Calibration 1 on full detectors

e.g. space charge distortion

QC

Detectors electronics

TPC TRD…Trigger

and clockITS …

Detector data samplesinterleaved with synchronized

heartbeat triggers

O2 architecture (2)


O2/T0/T1

EPNs


T0/T1

ArchiveStorage

Condition & Calibration Database

Quality Control

Sub-timeframesTimeframesCompressed timeframesAOD

CCDB Objects

Asy

nch

ron

ou

sSy

nch

ron

ou

s

QC data

CTFAOD

Storage


O2/T0/T1 O(1)

ESD, AOD

O2/T0/T1 O(1)

Event extractionTagging

Globalreconstruction

QCAOD extraction

Calibration 2

O2 architecture (3)


O2/T0/T1

Reconstructionpasses

and event extraction


T0/T1

Archive

Analysis

Storage

Simulation

Asy

nch

ron

ou

s

CTFAOD

Analysis Facilities

StorageHistograms,trees

O(1)

Analysis

AOD

Storage

T2

Simulation

CTF

AOD O(10)

QCReconstructionEvent buildingAOD extraction

ESD, AOD

Event Summary DataAnalysis Object Data

ESD, AODCompressed timeframes

Physics software designProcessing workflow


EPN: synchronous asynchronousAll FLPs

Raw data

Local Processing E.g.

ClusterizationCalibration

Detector ReconstructionE.g. TPC & ITSTrack finding

CTF AOD

Step 1 Step 2 Step 3 Step 4

Inter-detectormatching

procedures

Final calibration, 2nd matching

Final matching, PID, Event extraction

Step 0

Computing requirements for processing


Computing requirements -> Total : ~ 100000 CPU cores 5000 GPU chips

Goes together, mergingand fitting can run on

GPUs too

Being ported to GPU, conversion factor

unknown

Theoretically could runon GPU

Physics programme and data taking scenariosChapter 2


ALICE running scenarios :Year System √sNN Lint Ncollisions

(TeV) (pb-1) (nb-1)

2020pp 14 0.4 2.7 · 1010

Pb-Pb 5.5 2.85 2.3 · 1010

2021pp 14 0.4 2.7 · 1010

Pb-Pb 5.5 2.85 2.3 · 1010

2022pp 14 0.4 2.7 · 1010

pp 5.5 6 4 · 1011

2025pp 14 0.4 2.7 · 1010

Pb-Pb 5.5 2.85 2.3 · 1010

2026

pp 14 0.4 2.7 · 1010

Pb-Pb 5.5 1.4 1.1 · 1010

p-Pb 8.8 50 1011

2027pp 14 0.4 2.7 · 1010

Pb-Pb 5.5 2.85 2.3 · 1010

Data types characteristics


▶ TF size - Duration of the time window (tTF)▶ Data lost at the edges: 0.1/tTF(ms)▶ For calibration and reconstruction: 20ms - 100ms▶ Shorter is better for buffering and distribution 20ms (1000 interactions in Pb-Pb at 50kHz)

Data type Size (GB) Tape copy

TF (Pb-Pb) 10 No

CTF (Pb-Pb) 1.6 Yes

ESD 15% of CTF No

AOD 10% of CTF Yes

MC 100% of CTF No

MCAOD 30% of ESD Yes

HISTO 1% of ESD No

O2 facility design (2)


Network layout 2 : 4 independent EPN subfarms

FLP1

4 x 10 Gb/s

NetworkSub-Farm 4

EPN

EPN

1126

1500

FLP256

NetworkSub-Farm 3

NetworkSub-Farm 2

NetworkSub-Farm 1

EPN

EPN

751

1125

EPN

EPN

376

750

EPN

EPN

1

375

10 Gb/s

…

…

…

…

…

O2 facility design (3)


FLPEPN

FLPEPN

25

1

1

30

140/56 Gb/s

SEPN

SEPN

110 Gb/s1

EPN

EPN

1471

1500

10 Gb/s50

FLP

FLP250

226

10

10 X 40/56 Gb/s

50

2 X 40/56 Gb/s

Network layout 3 : Super-EPNs

O2 facility design (4)Simulation – Link speed


Left : Network Layout 2 : Link speed on the FLPs and EPNs for a network layout with 4 EPN subfarms for 100 parallel transfers from the FLPs.

Right : Network Layout 3 : Link speed on the FLPs and Super-EPNs(configuration based on an Infiniband network at 56 Gb/s)

O2 facility design (6)Simulation - system scalability


Latency of the timeframes for different interaction rates using layout 2 (left) and layout 3 (right) Layout 2 is cheaper but scales up to 90kHz only.

0 50 100 150 2005

10

15

20

25

30

35

40

45

50

55

Time [s]

La

ten

cy [

s]

50 kHz

80 kHz

90 kHz

0 20 40 60 80 100 120 1401.4

1.5

1.6

1.7

1.8

1.9

2

2.1

2.2

Time [s] L

ate

ncy [

s]

50 kHz

100 kHz

140 kHz

O2 facility – Power and cooling


Network performance tests


Comparison of Ethernet, IP over InfiniBand and IP over Omni-Path

40 GbE: 40 Gigabit EthernetOPA: Intel® Omni-Path

IB: InfiniBandO2: ALICE Online-Offline framework

0

20

40

60

80

100

Ne

two

rk T

hro

ugh

pu

t (G

b/s

)

Block Size (B)

40 GbE (O2) 40 GbE (max) OPA (O2)

OPA (max) IB (O2) IB (max)

1K 10K 100K 1MB 10M 50M0

10

20

30

40

50

Ne

two

rk T

hro

ugh

pu

t (G

b/s

)

Block Size (B)

40 GbE (O2) 40 GbE (max)

Saturating the Senderwith Ethernet

1K 10K 100K 1MB 10M 50M

Documents

ALICE trigger and computing upgrade: design, technology