Kostas KORDAS
INFN – Frascati
XI Bruno Touschek spring school, Frascati,19 May 2006
Higgs → 2e+2O(1/hr)
Higgs → 2e+2O(1/hr)
~25 min bias events ( >2k particles )
every 25 ns
~25 min bias events ( >2k particles )
every 25 ns
ATLAS
Trigger & Data Acquisition
system:
concept & architecture
ATLAS
Trigger & Data Acquisition
system:
concept & architecture
XI Bruno Touschek school, Frascati, 19 May '06
ATLAS TDAQ concept & architecture - Kostas KORDAS 2
LHCTeVatron
Process (pb) N/s N/year
Total collected before start of LHC
W l 3104 30 108 104 LEP / 107 FNAL
Z ee 1.5103 1.5 107 107 LEP
t t 830 1 107 104 Tevatron
b b 5108 106 1013 109 Belle/BaBar ?
Low lumi = 10 fb-1/y
ATLAS Trigger & DAQ: the need (1)ATLAS Trigger & DAQ: the need (1)
Total cross section is at ~100 mb, While the very interesting physics is at
~1 nb to ~1 pb,i.e., a ratio of 1:108 to 1:1011
XI Bruno Touschek school, Frascati, 19 May '06
ATLAS TDAQ concept & architecture - Kostas KORDAS 3
ATLAS Trigger & DAQ: the need (2)ATLAS Trigger & DAQ: the need (2)
40 MHz
~ 200 Hz ~ 300 MB/s
Full info / event: ~ 1.6 MB/25ns ~60k TB/s
p p
Need high luminosity to get to observe the
very interesting events
Need on-line selection to write to disk
mostly interesting events
XI Bruno Touschek school, Frascati, 19 May '06
ATLAS TDAQ concept & architecture - Kostas KORDAS 4
Dataflow
EBHigh LevelTrigger
L2
ROS
Level 1Det.
R/O
Trigger DAQ
2.5 s
~10 ms
Calo MuTrCh Other detectors
Read-Out Systems
L2P L2N
RoI
RoI data (~2%)
RoI requests
L2 accept (~3.5 kHz)
SFO
L1 accept (100 kHz)
40 MHz
EFEFP
~ sec
EF accept (~0.2 kHz)
ROD ROD ROD
ROB ROB ROB
SFI
EBN
Event Builder
EFN
DFM
L2SVROIB
Event Filter
Level 2
ATLAS Trigger & DAQ: architectureATLAS Trigger & DAQ: architecture
40 MHz
~ 200 Hz ~ 300 MB/s
100 kHz
~ 3.5 kHz
Full info / event: ~ 1.6 MB/25ns
~3+6 GB/s
160 GB/s
XI Bruno Touschek school, Frascati, 19 May '06
ATLAS TDAQ concept & architecture - Kostas KORDAS 5
From the detector into the Level-1 TriggerFrom the detector into the Level-1 Trigger
• Interactions every 25 ns:
…in 25 ns particles travel 7.5 m• Cable length ~100 meters:
…in 25 ns signals travel 5 m
Total Level-1 latency = 2.5 sec(TOF + cables + processing + distribution)
For 2.5 sec, all signals must be stored in electronic pipelines
Weight: 7000 t
44 m
22m
Level 1
Trigger DAQ
2.5 s
Calo MuTrChOther detectors
FE Pipelines
40 MHz
40 MHz
XI Bruno Touschek school, Frascati, 19 May '06
ATLAS TDAQ concept & architecture - Kostas KORDAS 6
Upon LVL1 accept: buffer data & get RoIsUpon LVL1 accept: buffer data & get RoIs
ROS
Level 1Det.
R/O
Trigger DAQ
2.5 s
Calo MuTrCh Other detectors
Read-Out Systems
RoI
L1 accept (100 kHz)
40 MHz
40 MHz
160 GB/sROD ROD ROD
ROB ROB ROBROIB
Read-Out Drivers
Region of Interest Builder Read-Out Buffers
Read-Out Links (S-LINK)
100 kHz
XI Bruno Touschek school, Frascati, 19 May '06
ATLAS TDAQ concept & architecture - Kostas KORDAS 7
LVL1 finds Regions of Interest for next levelsLVL1 finds Regions of Interest for next levels
4 RoI addresses
In this example:4 Regions of Interest:
2 muons,2 electrons
XI Bruno Touschek school, Frascati, 19 May '06
ATLAS TDAQ concept & architecture - Kostas KORDAS 8
Upon LVL1 accept: buffer data & get RoIsUpon LVL1 accept: buffer data & get RoIs
ROS
Level 1Det.
R/O
Trigger DAQ
2.5 s
Calo MuTrCh Other detectors
Read-Out Systems
RoI
L1 accept (100 kHz)
40 MHz
40 MHz
160 GB/sROD ROD ROD
ROB ROB ROBROIB
Read-Out Drivers
Region of Interest Builder Read-Out Buffers
Read-Out Links (S-LINK)
100 kHz
On average, LVL1 finds~2 Regions of Interest (in ) per event
Data in RoIs is a few % of the Level-1 throughput
XI Bruno Touschek school, Frascati, 19 May '06
ATLAS TDAQ concept & architecture - Kostas KORDAS 9
LVL2: work with “interesting” ROSs/ROBsLVL2: work with “interesting” ROSs/ROBs
For each detector there is a simple correspondence Region Of Interest ROB(s)
LVL2 Proccessing Units:
for each RoI, the list of ROBs with the corresponding data from each detector is quickly identified
L2
ROS
Level 1Det.
R/O
Trigger DAQ
2.5 s
~10 ms
Calo MuTrCh Other detectors
Read-Out SystemsL2P L2N
RoI
RoI data (~2%)
RoI requests
L1 accept (100 kHz)
40 MHz
40 MHz
100 kHz 160 GB/s
~3 GB/s
ROD ROD ROD
ROB ROB ROBL2SVROIBLevel 2
LVL2 Supervisor
LVL2 Network
LVL2 Processing Units
Read-Out Buffers
RoI-based Level-2 trigger:A much
smaller ReadOut network
… at the cost of a higher control traffic
XI Bruno Touschek school, Frascati, 19 May '06
ATLAS TDAQ concept & architecture - Kostas KORDAS 10
Trigger DAQCalo
MuTrCh
EB
L2
ROS
Level 1Det.
R/O
2.5 s
~10 ms
Other detectors
Read-Out Systems
L2P L2N
RoI
RoI data (~2%)
RoI requests
L2 accept (~3.5 kHz)
L1 accept (100 kHz)
40 MHz
40 MHz
100 kHz
~3.5 kHz
160 GB/s
~3+6 GB/s
ROD ROD ROD
ROB ROB ROB
SFI
EBN
Event Builder
DFM
L2SVROIBLevel 2
Sub-Farm Input
Dataflow ManagerEvent Building Network
After LVL2: Build full eventsAfter LVL2: Build full events
XI Bruno Touschek school, Frascati, 19 May '06
ATLAS TDAQ concept & architecture - Kostas KORDAS 11
LVL3: Event Filter deals with Full Event infoLVL3: Event Filter deals with Full Event infoTrigger DAQ
EB
L2
ROS
Level 1Det.
R/O
2.5 s
~10 ms
Calo MuTrCh Other detectors
Read-Out SystemsL2P L2N
RoI
RoI data (~2%)
RoI requests
L2 accept (~3.5 kHz)
L1 accept (100 kHz)
40 MHz
40 MHz
100 kHz
~3.5 kHz
160 GB/s
~3+6 GB/s
EFEFP
~ sec
ROD ROD ROD
ROB ROB ROB
SFI
EBN Event Builder
EFN
DFM
L2SVROIB
Event Filter
Level 2
Farm ofEvent Filter Processors
Event Filter Network
Full Event Sub-Farm Input
~ 200 Hz
XI Bruno Touschek school, Frascati, 19 May '06
ATLAS TDAQ concept & architecture - Kostas KORDAS 12
EB
L2
ROS
Level 1Det.
R/O
2.5 s
~10 ms
Calo MuTrCh Other detectors
Read-Out SystemsL2P L2N
RoI
RoI data (~2%)
RoI requests
L2 accept (~3.5 kHz)
L1 accept (100 kHz)
40 MHz
40 MHz
100 kHz
~3.5 kHz
160 GB/s
~3+6 GB/s
EFEFP
~ sec
ROD ROD ROD
ROB ROB ROB
SFI
EBN
Event Builder
EFN
DFM
L2SVROIB
Event Filter
Level 2
Event Filter Processors
Event Filter Network
SFOEF accept (~0.2 kHz)
~ 200 Hz ~ 300 MB/s
Sub-Farm Output
From Event Filter to Local (TDAQ) storageFrom Event Filter to Local (TDAQ) storage
XI Bruno Touschek school, Frascati, 19 May '06
ATLAS TDAQ concept & architecture - Kostas KORDAS 13
Dataflow
EBHigh LevelTrigger
L2
ROS
Level 1Det.
R/O
Trigger DAQ
2.5 s
~10 ms
Calo MuTrCh Other detectors
Read-Out Systems
L2P L2N
RoI
RoI data (~2%)
RoI requests
L2 accept (~3.5 kHz)
SFO
L1 accept (100 kHz)
40 MHz
40 MHz
100 kHz
~3.5 kHz
~ 200 Hz
160 GB/s
~ 300 MB/s
~3+6 GB/s
EFEFP
~ sec
EF accept (~0.2 kHz)
ROD ROD ROD
ROB ROB ROB
SFI
EBN
Event Builder
EFN
DFM
L2SVROIB
Event Filter
Level 2
TDAQ, High Level Trigger & DataFlowTDAQ, High Level Trigger & DataFlow
XI Bruno Touschek school, Frascati, 19 May '06
ATLAS TDAQ concept & architecture - Kostas KORDAS 14
Dataflow
EBHigh LevelTrigger
L2
ROS
Level 1Det.
R/O
Trigger DAQ
2.5 s
~10 ms
Calo MuTrCh Other detectors
Read-Out Systems
L2P L2N
RoI
RoI data (~2%)
RoI requests
L2 accept (~3.5 kHz)
SFO
L1 accept (100 kHz)
40 MHz
40 MHz
100 kHz
~3.5 kHz
~ 200 Hz
160 GB/s
~ 300 MB/s
~3+6 GB/s
EFEFP
~ sec
EF accept (~0.2 kHz)
ROD ROD ROD
ROB ROB ROB
SFI
EBN
Event Builder
EFN
DFM
L2SVROIB
Event Filter
Level 2
TDAQ, High Level Trigger & DataFlowTDAQ, High Level Trigger & DataFlow
High Level Trigger (HLT)
• Algorithms developed offline (with HLT in mind)
• HLT Infrastructure (TDAQ job):– “steer” the order of
algorithm execution– Alternate steps of “feature
extraction” & “hypothesis testing”)
fast rejection (min. CPU)
– Reconstruction in Regions of Interest min. processing time &
network resources
XI Bruno Touschek school, Frascati, 19 May '06
ATLAS TDAQ concept & architecture - Kostas KORDAS 15
Dataflow
EBHigh LevelTrigger
L2
ROS
Level 1Det.
R/O
Trigger DAQ
2.5 s
~10 ms
Calo MuTrCh Other detectors
Read-Out Systems
L2P L2N
RoI
RoI data (~2%)
RoI requests
L2 accept (~3.5 kHz)
SFO
L1 accept (100 kHz)
40 MHz
40 MHz
100 kHz
~3.5 kHz
~ 200 Hz
160 GB/s
~ 300 MB/s
~3+6 GB/s
EFEFP
~ sec
EF accept (~0.2 kHz)
ROD ROD ROD
ROB ROB ROB
SFI
EBN
Event Builder
EFN
DFM
L2SVROIB
Event Filter
Level 2
TDAQ, High Level Trigger & DataFlowTDAQ, High Level Trigger & DataFlow
DataFlow
• Buffer & serve data to HLT
• Act according to HLT result, but otherwise HLT is a “black box” which gives answers
• Software framework based on C++ code and the STL
XI Bruno Touschek school, Frascati, 19 May '06
ATLAS TDAQ concept & architecture - Kostas KORDAS 16
Dataflow
EBHigh LevelTrigger
L2
ROS
Level 1Det.
R/O
Trigger DAQ
2.5 s
~10 ms
Calo MuTrCh Other detectors
L2P L2N
RoI
RoI data (~2%)
RoI requests
L2 accept (~3.5 kHz)
SFO
L1 accept (100 kHz)
40 MHz
EFEFP
~ sec
EF accept (~0.2 kHz)
ROD ROD ROD
ROB ROB ROB
SFI
EBN
EFN
DFM
L2SVROIB500nodes
100nodes
150nodes
1600nodes
Infrastructure Control Communication Databases
High Level Trigger & DataFlow: PCs (Linux)High Level Trigger & DataFlow: PCs (Linux)
XI Bruno Touschek school, Frascati, 19 May '06
ATLAS TDAQ concept & architecture - Kostas KORDAS 17
TDAQ at the ATLAS siteTDAQ at the ATLAS site
SDX1
USA15
UX15
ATLASdetector
Read-Out
Drivers(RODs) First-
leveltrigger
Read-OutSubsystems
(ROSs)
UX15
USA15
Dedicated links
Timing Trigger Control (TTC)
1600Read-OutLinks
Gig
abit
Eth
erne
t
RoIBuilder
Reg
ions
Of I
nter
est
VME~150PCs
Data of events acceptedby first-level trigger
Eve
nt d
ata
requ
ests
Del
ete
com
man
ds
Req
uest
ed e
vent
dat
a
Event data pushed @ ≤ 100 kHz, 1600 fragments of ~ 1 kByte each
LVL2Super-visor
SDX1CERN computer centre
DataFlowManager
EventFilter(EF)
pROS
~ 500 ~1600
stores LVL2output
dual-CPU nodes
~100 ~30
Network switches
Event data pulled:partial events @ ≤ 100 kHz, full events @ ~ 3 kHz
Event rate ~ 200 HzData
storage
LocalStorage
SubFarmOutputs
(SFOs)
LVL2 farm
Network switches
EventBuilder
SubFarmInputs
(SFIs)
Second-leveltrigger
“pre-series” system: ~10% of final TDAQ
in place
XI Bruno Touschek school, Frascati, 19 May '06
ATLAS TDAQ concept & architecture - Kostas KORDAS 18
Example of worries in such a system: CPU powerExample of worries in such a system: CPU power
• At Technical Design Report we assumed:
– 100 kHz LVL1 accept rate– 500 dual-CPU PCs for LVL2– 8 GHz per CPU at LVL2
• So:– each L2PU does 100Hz– 10ms average latency per event in each L2PU
• 8 GHz per CPU will not come– But, dual-core dual-CPU PCs show scaling!
Scaling of a dual-core dual-CPU Processor
0
100
200
300
400
500
1 2 3 4 5 6
Number of Processes
Ach
ieved
LV
L2 R
ate
(H
z)
Series1
Preloaded ROS w/ muon events, run muFast @ LVL2
Test with AMD dual-core, dual CPU @ 1.8 GHz, 4 GB total
We should reach necessary performance per PC at cost of higher memory needs & latency (shared memory model would be better here)
18
XI Bruno Touschek school, Frascati, 19 May '06
ATLAS TDAQ concept & architecture - Kostas KORDAS 19
Last Sept: cosmics in the Tile hadronic calorimeter, brought via the pre-series (monitoring algorithms)
Cosmics in ATLAS in the pitCosmics in ATLAS in the pit
This July: Cosmic run with
LAr EM + Tile Had Cal (+Muon detectors?)
XI Bruno Touschek school, Frascati, 19 May '06
ATLAS TDAQ concept & architecture - Kostas KORDAS 20
• ATLAS TDAQ:– 3-level trigger hierarchy– Use Regions of Interest from previous level: small data movement– Feature extraction + hypothesis testing: fast rejection min. CPU
power
SummarySummary
• We are in the installation phase of system• Cosmic run with Central Calorimeters (+muon system?) this summer
TDAQ will be ready in time
for LHC data taking
• Triggering at Hadron Colliders:– Need high luminosity to get rare events
– Can not write all data to disk• No sense otherwise: offline, we’ll be wasting our time looking for a needle in
the hay!
XI Bruno Touschek school, Frascati, 19 May '06
ATLAS TDAQ concept & architecture - Kostas KORDAS 21
Thank you
XI Bruno Touschek school, Frascati, 19 May '06
ATLAS TDAQ concept & architecture - Kostas KORDAS 22
ROS units contain 12 R/O Buffers150 units needed for ATLAS (~1600 ROBs)
A ROS unit is implemented with a 3.4 GHz PC housing 4 custom PCI-x cards (ROBIN)
ReadOut Systems: 150 PCs w/ special cardsReadOut Systems: 150 PCs w/ special cards12 ROS in place, more arriving
Performance of final ROS (PC+ROBIN)
is above requirements
Note: we have also ability to access individual ROBs if
wanted/needed50
60
70
80
90
100
110
120
130
140
0 2 4 6 8 10
Level 2 Trigger acceptance (%)
Le
ve
l 1
Tri
gg
er
rate
(kH
z)
“Hottest” ROS from paper model
2. Measurements on real ROS H/W
Low Lumi. operating region
High Lumi.
operating
region
LV
L1
acce
pt
rate
(kH
Z)
LVL2 accept rate (% of input)
Not all ROSs are equal in rate of data requestsRODROS re-mapping can reduce requirements on busiest (hottest) ROS
XI Bruno Touschek school, Frascati, 19 May '06
ATLAS TDAQ concept & architecture - Kostas KORDAS 23
So, we need:• 5600 MB/s into EB system / (70MB/s in each EB node)
need ~80 SFIs for full ATLAS • When SFI serves EF, throughput decreases by ~20%
actually need 80/0.80 = 100 SFIs
Event Building needsEvent Building needsThroughput requirements:
• 100 KHz LVL1 accept rate• 3.5% LVL2 accept rate 3.5 KHz EB• 1.6 MB event size 3.5 x 1.6 = 5600 MB/s total input
Network limited (fast CPUs):• Event building using 60-70% of Gbit network ~70 MB/s into each Event Building node (SFI)
6 prototypes in place, evaluation of PCs now, expect big Event Building needs from day 1: > 50 PCs till end of year
XI Bruno Touschek school, Frascati, 19 May '06
ATLAS TDAQ concept & architecture - Kostas KORDAS 24
Data File
LVL2 Ltcy
ProcessTime
RoI CollTime
RoI Coll Size
# Req /Evt
(ms) (ms) (ms) (bytes)
3.4 2.8 0.6 287 1.3
di-jet 3.6 3.3 0.3 2785 1.2
e 17.2 15.5 1.7 15820 7.4
Tests of LVL2 algorithms & RoI collection Tests of LVL2 algorithms & RoI collection
2) Processing takes ~all latency:
small RoI data collection time
Note: Neither Trigger menu, nor data files representative mix of ATLAS (this is the aim for a late 2006 milestone)
3) Small RoI data request
per event
Electron sample
is pre-selected
1) Majority of events rejected
fast
Di-jet, & e simulated events preloaded on ROSs; RoI info on L2SV
L2SV
L2PU
pROSEmulated
ROS
8
1
1
pROS1
DFM
1
Plus:• 1 Online Server• 1 MySQL data base server
XI Bruno Touschek school, Frascati, 19 May '06
ATLAS TDAQ concept & architecture - Kostas KORDAS 25
• L2SV gets RoI info from RoIB• Assigns a L2PU to work on event• Load-balances its’ L2PU sub-farm
• Can scheme cope with LVL1 rate?• Test with preloaded RoI info into RoIB, which triggers TDAQ chain, emulating LVL1
• LVL2 system is able to sustain the LVL1 input rate:– 1 L2SV system for LVL1 rate ~ 35 kHz– 2 L2SV system for LVL1 rate ~ 70 kHz (50%-50% sharing)
Scalability of LVL2 system Scalability of LVL2 systemRoIB -> 1 or 2 L2SVs.
Each L2SV->1-8L2PUs
34
35
36
1 3 5 7
# L2PUs
LV
L1
ra
te (
KH
z)
1L2SV-1
2L2SV-1
2L2SV-2
Rate per L2SV stable within 1.5%
ATLAS will have a handful of L2SVs can easily manage 100 kHz LVL1 rate
XI Bruno Touschek school, Frascati, 19 May '06
ATLAS TDAQ concept & architecture - Kostas KORDAS 26
• Previous Event Filter I/O protocol limited rate for small event sizes (e.g., partially built)
changed in current TDAQ software release
EF performance scales farm sizeEF performance scales farm sizeDummy algorithm: always accept,
but with fixed delay
Event size1 MB
Initially CPU limited, but eventually bandwidth limited
• Test e/ & selection algorithms– HLT algorithms seeded by L2Result– pre-loaded (e & ) simulated events
on 1 SFI Emulator serving EF farm– Results here are for muons:
Running muon algorithms: scaling with EF farm size (still CPU limited with 9 nodes)
XI Bruno Touschek school, Frascati, 19 May '06
ATLAS TDAQ concept & architecture - Kostas KORDAS 27
ATLAS Trigger & DAQ: philosophy ATLAS Trigger & DAQ: philosophy
40 MHz
~100 kHz
2.5 s
~3 kHz
~10 ms
~ 1 s
~200 Hz
Muon
LVL1
Calo Inner
PipelineMemories
Read-OutDrivers
RatesLatency
RoI
LVL2
Event builder cluster
Local Storage: ~ 300 MB/s
Read-Out Subsystems
hosting Read-Out
Buffers
Event Filter farm
EF
ROBROBROBROBROBROBROBROBROBROBROBROB
RODRODRODRODRODROD
RODRODROD
ROBROBROBROBROBROBROBROBROBROBROBROB
Hardware based (FPGA, ASIC)Hardware based (FPGA, ASIC)
Calo/Muon (coarse granularity)Calo/Muon (coarse granularity)
Software (specialised algs)Software (specialised algs)
Uses LVL1 Uses LVL1 Regions of InterestRegions of Interest
AllAll sub-dets, sub-dets, fullfull granularity granularity
Emphasis on early rejectionEmphasis on early rejection
Offline algorithmsOffline algorithms
Seeded by Seeded by LVL2 resultLVL2 result
Work with Work with full eventfull event
Full calibration/alignment infoFull calibration/alignment info
Hig
h L
evel Tri
gg
er
Hig
h L
evel Tri
gg
er
XI Bruno Touschek school, Frascati, 19 May '06
ATLAS TDAQ concept & architecture - Kostas KORDAS 28
L2PU
pROS
(EB node)
RoIB
L2SV
ROS
LVL1 Trigger
EFD/PT (EF node)
4: RoI Request
5: RoI Data
6: L2Result
7: Ack
16: Full Event
12: Data Request
15: Clear event
13: ROS Data
2: L1Result3: L1Result
8: L2Decision
9: L2DescisionGroup
10: Ack
1: L1 Trigger streams
15: Clear event
13: ROS Data
DFM
11: AssignmentSFI
12: Data Request
14: End Of Event
SFO
17: Full Event
L2PU
pROS
(EB node)
RoIB
L2SV
ROS
LVL1 Trigger
EFD/PT (EF node)
4: RoI Request
5: RoI Data
6: L2Result
7: Ack
16: Full Event
12: Data Request
15: Clear event
13: ROS Data
2: L1Result3: L1Result
8: L2Decision
9: L2DescisionGroup
10: Ack
1: L1 Trigger streams
15: Clear event
13: ROS Data
DFM
11: AssignmentSFI
12: Data Request
14: End Of Event
SFO
17: Full Event
Data Flow and Message PassingData Flow and Message Passing
XI Bruno Touschek school, Frascati, 19 May '06
ATLAS TDAQ concept & architecture - Kostas KORDAS 29
A Data Collection application example: the Event Builder
A Data Collection application example: the Event Builder
Event Assembly Activity
Input Activity
Request Activity Event Handler ActivityAssignment
*EventFragments *Event
Event Sampler Activity
ROS &
pROSEvent Fragments Data Requests
Data Flow Manager
Assignment
Event
Event
Trigger(EventFilter)
EventMonitoring
SFI:Event
Builder