View
217
Download
0
Category
Preview:
Citation preview
Level-1 Calorimeter TriggerReadout Driver FDR/PRR
15th August 2006
Introduction to the ROD
Norman Gee 15-Aug-2006
Norman Gee
15 Aug 2006 Norman Gee - L1Calo ROD Introduction 2
Contents
• Introduction
– Overview; Readout Scheme; Crate Structure; Physical Implementation
• Input and Output Data
– Characteristics of the data; Zero suppression / compression; Example data;
• Requirements summary
• Input data handling
– Readout process; Timeout; Rod Busy; TrigType Timeout; ECReset
• Output Data handling
– RoI Data; Data routing to S-Link; Data Reduction; Replay mode; Monitoring; generation of S-Link packets
• VME Interface, Canbus
15 Aug 2006 Norman Gee - L1Calo ROD Introduction 3
DAQRODs To ROS
e/, /hadClusters
(CP)
0.2 x 0.2Jet / ET
(JEP)
0.1 x 0.1
Pre-Processor
(PPr)Analogue
tower sums0.1 x 0.1(~7200)
RoIRODs To RoIB
To CTP
To CTP
8 PPr crates
4 CP crates
2 JEP crates
2 ROD crates
L1Calo Overview
DAQ
RoIs
15 Aug 2006 Norman Gee - L1Calo ROD Introduction 4
Processing Module Readout Scheme
FIFO
Glink
Serial Linkto ROD
Dual-port Memory
Shift Registers
40 MHz Processing Module Pipeline
FIFO FIFO
Dual-port MemoryDual-port Memory
L1A
Readout_scheme 03Aug06.cnv
Write
Read
ScrollingMemory
15 Aug 2006 Norman Gee - L1Calo ROD Introduction 5
CPU
Preprocessor Crate (1 of 8)
16 PPMsTCM
Disp
RODCrate
(1 of 2)
CPU
CMM
CMM
16 JEMs
Jet/Energy-sum Crate (1 of 2)
RoIsSlice Data
TCM
G-link
S-link
Slice Data to ROS(see text for discussion of maximum
number of S-Links)
CPM Crate (1 of 4)
RoIsSlice Data
CPU
TCM
CMM
CMM
14 CPMs
RODLINK 28Jul03
RoIs toROIB
CPU
PPM
ROD
PPM
ROD
PPM
ROD
PPM
ROD
CP
ROD
CP
ROD
JE
ROD
CProi
ROD
CProi
ROD
JEroi
ROD
TCM
From otherROD Crate
Slice Data
From three otherPreprocessor Crates
From anotherCPM Crate
From Muontrigger
and CTP
15 Aug 2006 Norman Gee - L1Calo ROD Introduction 6
Physical Implementation
• 9U * 400 mm Main module
• 9U * 160mm Rear module
DA
Q o
r R
oI d
ata
Rea
d-ou
t dat
from
up
to 1
8m
odul
es
P1
P2
P3
P0
RJ2
RJ3
RJ0
J1
J2
J3
J0
S-Link 0
S-Link 2
S Link 3
S-Link 1
RP2
RP3
GL
GL
GL
GL
GL
GL
GL
GL
GL
GL
GL
GL
GL
GL
GL
GL
GL
GL
VME
S-Link
Swtch
Receive data, Parity check andCompression
+Data Buffering
Receive data, Parity check andCompression
+Data Buffering
RodOverview 25Apr06.vsd
15 Aug 2006 Norman Gee - L1Calo ROD Introduction 7
Internal Structure
• 18 G-Links
– handled in groups of 4
• Five Input FPGAs
– data input, compression, formatting
• Switch FPGA
– Data routing to 4 S-Links
• Monitoring FPGA
– Simple event monitoring
• Overall module functions
– VME FPGA, ACE, CPLD, TTC,
– CANbus, power (details in Viraj’s talk)
Input FPGAReceive data, parity check and
compression+
Data BufferingXC2VP20-5FF896
32 + 4
G-Link
G-Link
G-Link
G-Link
96
32 + 4
G-Link
G-Link
G-Link
G-Link
96
32 + 4
G-Link
G-Link
G-Link
G-Link
96
32 + 4
G-Link
G-Link
G-Link
G-Link
96
32 + 4G-Link
G-Link48
Switch,Controller,
etc,
XC2VP30-5FF896
VME64XJ2
VME64XJ3
S-Link 0
S-Link 1
S-Link 2
S-Link 3
TTCrx
TTCrxDecoder
Card
XC2VP20-5FF896
VME64xInterface
Header
TCM
VM
E
CompactFLASH/IBMmicrodrive
ACEController
JTAG
System ACE
JTAG
SwitchFPGA
Spy EventBuffer
CANBus
Clock fanout
46
46
46
46
Pow
erP
CP
ower
PC
Monitor FPGA
Ins RAM(64Kx32)
Ins RAM(64Kx32)
Dual Port RAM(64K x 32)
PCIInterface
(33MHz,64bit)
MPUInterface
XC2VP20-5FG896
BCID NumberBuffer
Event NumberBuffer
Trailer
Trigger TypeBuffer
Status
Data Count
Dual OpticalReceivers
Dual OpticalReceivers
Dual OpticalReceivers
Dual OpticalReceivers
Dual OpticalReceivers
Dual OpticalReceivers
Dual OpticalReceivers
Dual OpticalReceivers
Dual OpticalReceivers
Links 0,1
Links 2,3
Links 4,5
Links 6,7
Links 8,9
Links 10,11
Links 12,13
Links 14,15
Links 16,17
Reset andModule
InitializationCPLD
System Reset
36
61
4 x Rocket I/O32
TTC
44
Input FPGAReceive data, parity check and
compression+
Data BufferingXC2VP20-5FF896
Input FPGAReceive data, parity check and
compression+
Data BufferingXC2VP20-5FF896
Input FPGAReceive data, parity check and
compression+
Data BufferingXC2VP20-5FF896
Input FPGAReceive data, parity check and
compression+
Data BufferingXC2VP20-5FF896
RodBlock 04Aug06.vsd
15 Aug 2006 Norman Gee - L1Calo ROD Introduction 8
Data Characteristics
• Preprocessor data has two components per channel (64 ch per module):
– Raw FADC output – typically ~five 11-bit values – to cover a calo pulse…
– …and a calibrated ET value in GeV/c (“LUT data”) .
– Most of the time, there is no calo pulse - so FADC output is pedestal + noise• Can reduce data volume by bit-field compression, zero-suppress LUT data.
Cannot discard FADC data (at least at low L1A rate) – needed to check BCID.
• Most other modules also produce two components, but very different from above:
– Integer values in GeV entering or leaving boards;
– Threshold hits – single bits to show if a digital value exceeds a threshold.• Zero suppression expected to be effective - calo cell value rarely exceeds 1 GeV• All information relating to one bunch crossing is read out in a single clock cycle.
• All the data reduction is designed to be lossless.
15 Aug 2006 Norman Gee - L1Calo ROD Introduction 9
Example G-Link Data format (JEM)
Jet Elements + Parity
GP
D0
Jet Elements + Parity
…
GP
D13
Jet Elements + Parity
GP
D14
Main Jet Counts
GP
EP
JP
D15
Energy Parity Jet Parity
Main & Fwd Jet Counts (Jems 0,7,8,15 only)
GP
EP
JP
D15
Energy Parity Jet Parity
Expanded jet element frame
LVDS Link Down LVDS Serialiser Error
04-Jan-05 Glinkfmt.xls/JEM DAQ
Jet Element data d8 ... d0SELD
2b F0
2b F1
2b F2
2b F3
8b Ex8b Ey8b Et2b M4
2b M5
2b M6
2b M7
2b M0
2b M1
2b M2
2b M3
12b BC Number4*0
11b Frame 3e
11b Frame 3e11b Frame 3h
3b Thr4
8b Ey4*0
11b Frame 2h
12b BC Number
11b Frame 1e11b Frame 1h
3b Thr5
3b Thr6
3b Thr7
11b Frame 2e
3b Thr1
3b Thr0
3b Thr2
3b Thr3
11b Frame 1e
11b Frame 1e11b Frame 1h
11b Frame 1h11b Frame 2h 11b Frame 2e
11b Frame 2e
8b Et 8b Ex
11b Frame 2h22*0
11b Frame 3h
15 Aug 2006 Norman Gee - L1Calo ROD Introduction 10
Example S-Link Data format
• Sub-Block Header
• Payload created from G-Link data
• Sub-block Trailer (suppressed if there were no errors)
• Note: there is also a “neutral” format. It copies raw data direct from G-Link to S-Link without interpretation. Very important for debugging.
Each JEM Sub-Block consists of the following:Fmt=1
Word Id L1Calo Sub-Block Header - always present
1 1 0 0
Word ID 0 or more trigger tower data words
0 1 0 P P zs
H parity error E parity error
PPm link down
Jet Element pair
Word ID 0 or 1 jet threshold data words (Main jets)
1 0 0 zs
Word ID 0 or 1 jet threshold data words (Main + Fwd jets)
1 0 0 zs
Word ID 0 or 1 energy subsum data words
1 0 1 zs
Word Id L1Calo SubStatus (omitted if no errors)
1 1 0 1GT
GD
DFO
BM
GI
GP
Failing BCN G-Link Timeout
G-Link down
DAQ Fifo Overflow
BCN Mismatch
G-Link Internal Protocol Error
G-Link Serial parity
12-Jan-05 Slinkfmt.xls/ROD JEM Datablock
6b BCN bits 5:0
3b Vers
4b Module
4b Crate
6b Seqno = Slice No
3b Fmt
2b PL
9b Electron data
4b N. Slices
4b NS2 ( 0 )
9b Hadron data
3b Thr4
3b Thr5
3b Thr6
3b Thr0
3b Thr1
3b Thr2
3b Thr3
5b Src Id (20)
5b Src Id (21)
2b JP
5b G-link Pin
2b F3
3b Thr7
8b Ey8b Et5b Src Id
(22)8b Ex
2b M7
2b M0
2b M1
2b M2
2b M3
2b F0
2b F1
2b F2
2*04b
Module4b
Crate6b Seqno
2b M4
2b M5
2b M6
15 Aug 2006 Norman Gee - L1Calo ROD Introduction 11
Summary of Requirements
• Receive 16- or 20-bit data from up to 18 G-Links. Check BCNs & parity.• Optionally discard some timeslices from different sections of source
modules.• Compress or zero suppress data from each channel, as required.• Write formatted data to the FIFO buffers (internal to the FPGAs)• Receive CLOCK, BCR, ECR, event number and event type from TTC.• Select data to be transferred to the ROS based on trigger type.• Build and write ATLAS-standard event fragments to up to four S-Links.• Spy on events (FPGA or VME), optionally select according to event type.• Generate BUSY signal to the Central Trigger Processor.• Process complete events at up to 100 KHz sustained rate.• For RoI data only, limit the number of RoIs to a programmable maximum.• Support CANBus board temperature and voltage monitoring.
15 Aug 2006 Norman Gee - L1Calo ROD Introduction 12
Description of readout process
• At Source Module
– L1A arrives, n (e.g. 5) timeslices copied into readout FIFOS in G-Link data format.
– DAV asserted. Data for n slices sent, with parity after each. DAV removed
• At the ROD
– L1A arrives, BCN stored in FIFOs in Input & Switch FPGAs
– Input FPGA: G-Link data received, reformatted, zero-suppressed, stored in data FIFO.
• Checks on G-Link status, Longitudinal parity, BCN -> error bits in substatus word
– Switch FPGA: receives event type.
– Switch FPGA: waits for all Input FPGAs to complete; creates S-Link header; copies payload to S-Link; creates S-Link trailer;
15 Aug 2006 Norman Gee - L1Calo ROD Introduction 13
Input data timeout
• Happens if a source module fails to provide expected data.
• Input timeout is detected by Switch FPGA:
– Start a timer when the first Input FPGA has processed data available.
– Declare Input timeout if timer expires before all expected Inpout FPGA channels have processed data available.
• When building the S-Link packet, timed-out Input channels are instructed to provide a sub-header and a sub-status word with timeout flag set.
– Late data is not used even if available by then.
• If the missing data does eventually arrive (unlikely), it will be put into the following event
– but will have wrong Bunch Number.
• Any surplus data is erased by the ECR mechanism.
15 Aug 2006 Norman Gee - L1Calo ROD Introduction 14
Generation of ROD- Busy
• Four S-Links - each can assert LFF (“Xoff”) if destination is too slow.
• When LFF, Switch stops working. Input FPGA Data FIFOs start to fill up.
• When depth limit is reached, ROD-BUSY is generated.
– Programmable depth limits on three FIFOs – event data, event management, BCN FIFOs.
– Must be set to handle worst case – 8 events separated by 5 bc each.
• We have two sets of counters:
– one set to monitor the fraction of time each S-Link is asserting LFF.
– one set as an 8-bin histogram of Input FPGA data FIFO depth.
15 Aug 2006 Norman Gee - L1Calo ROD Introduction 15
Trigger-type timeout
• Trigger type is sent by TTC broadcast so has no guaranteed arrival time
– depends on other TTC channel B traffic.
• Trigger type is needed to populate ROD header.
• If ROD waits indefinitely, Input FPGA FIFOs will fill up and ROD BUSY will be generated.
– If trigger type doesn’t ever arrive, ROD will block.
• Trigger-type timeout starts once processed data for an event is ready in the Input FPGAs.
• After the timeout, a default value will be used.
• If trigger type arrives very late, it is cleared by ECR
15 Aug 2006 Norman Gee - L1Calo ROD Introduction 16
Event Counter Reset
• Sent by CTP at about 1 Hz. Ignored by TTCrx ASIC, but used in ROD to clear ROD-internal 24-bit L1_ID and increment 8-bit ECR Counter.
• Both together form 32-bit ROD_L1ID field in S-Link header.
• Also usedin L1Calo RODs to clear any partial events.
ECR Deadtime
L1AWorst Case
S-Link Readout
ECR
Buffer Reset
ECRTiming 30Jul03
15 Aug 2006 Norman Gee - L1Calo ROD Introduction 17
RoI Data
• Collected from source modules just like DAQ data
– One timeslice only.
• Sent to separate “RoI-RODS” on separate links. Zero-suppressed as DAQ data.
• Collected by Switch FPGA and counted as sent to RoIB over S-link
• Programmable limit to number of RoIs sent.
– When this limit is reached, no more RoI payload is sent. Status bit is set.
– This guarantees that RoIB is not overwhelmed by wild events in ATLAS
• Normally only 1-2 RoIs expected per event.
– Calibration runs will produce many more – not used in Level-2.
15 Aug 2006 Norman Gee - L1Calo ROD Introduction 18
Data Routing to S-links – DAQ Data
• Data from each Input FPGA (4 G-Links) is concatenated into a separate ROD fragment. Data from the last two input FPGAs (nos 3 & 4) are combined into one fragment
– G-links 0 - 3 -> S-link 0
– G-Links 4 – 7 -> S-link 1
– G-Links 8-11 -> S-Link 2
– G-Links 12-17 -> S-Link 3
• All data from Input FPGAs 3 & 4 (4 or 6 G-Links) is concatenated into S-Link 3.
S-Link 0
S-Link 1
S-Link 2
1
2
3
4
5
6
7
8
9
10
11
12
13
14
DAQ
DAQ
15
16FPGA
+Data
Buffers
S-Link 3DAQ
0-3
4-7
8-11
12-15
ROD - Preprocessor, CP, JE Slice Data S-Link RTM
SwitchSwitch
Switch
FPGA+
DataBuffers
FPGA+
DataBuffers
FPGA+
DataBuffers
FPGA+
DataBuffers
17
S-Link DAQ 25Apr06
DAQ
0
16-17
15 Aug 2006 Norman Gee - L1Calo ROD Introduction 19
Data Routing to S-links – RoI Data
• All data from 18 G-Links is concatenated into a single ROD fragment
• The fragment is duplicated to Slinks 0 and 2.
– S-Link 2 has a logic analyser header.
S-Link 0
S-Link 1
S-Link 2
17
0
FPGA+
DataBuffers
DAQ
RoI
FPGA+
DataBuffers
S-Link 3
0-3
4-7
8-11
12-15
ROD -RoI Data S-Link RTM
Switch
Switch
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
FPGA+
DataBuffers
FPGA+
DataBuffers
FPGA+
DataBuffers
Slink_RoI_26Apr06
16-17
15 Aug 2006 Norman Gee - L1Calo ROD Introduction 20
Data Reduction
• Most data (apart from PPM) is subject to zero-suppression. S-Link words are not written if the data values and error bits are all zero.
– Typically expect only a few percent of non-zero trigger towers.
– For 5% occupancy of trigger towers, achieve factor 5-10 reduction
• Addressing information is added to identify remaining data:
– in ROD header (identifies DAQ and RoI, plus crate family)
– in Sub-block header (identifies source crate, type of module, slice number)
– in S-Link words (identify data source within module)
• For the PPM, zero-suppression is not appropriate, so a variable-length bit field approach is used.
– Based on subtracting FADC samples to give small numbers, which can be packed in short bit-fields. Expect a factor 2 data reduction.
– Prototype scheme is documented and checked. Will be tuned.
15 Aug 2006 Norman Gee - L1Calo ROD Introduction 21
Replay Mode
• Replay mode runs from the Input FPGAs.
– Disable all G-Links;
– Preload FIFOs with required data and put lengths in event management FIFOs.
– Generate L1As internally (software) or using TTC system.
– Transmitted data is recycled back into the FIFOs so one or several events can be played continuously
• BCN, Event number and Trigger Type are obtained from fixed registers, not TTC.
15 Aug 2006 Norman Gee - L1Calo ROD Introduction 22
Event Sampling and Monitoring
• Event sampling buffers are provided in the Switch FPGA for each S-Link. Sampled events may be accessed from VME or copied to the Monitoring FPGA.
• Monitoring FPGA code is not yet written. Data will be available for monitoring by code in FPGA-internal PC cores or to PCI daughterboard.
• Events are eligible for sampling if they satisfy an event-type mask and a prescale factor.
– All S-Links are sampled (or not) on the same event.
• Two sampling modes:
– Random: events sampled whenever a buffer is available;
– 100%: All events sampled. Software LFF is asserted if all buffers are full. This mode will assert ROD-BUSY if the trigger rate is too high.
• A mechanism is provided to sample the same events in all RODs
15 Aug 2006 Norman Gee - L1Calo ROD Introduction 23
Generation of S-Link fragments
• 5 components:
– BOF and EOF control words• Fixed in Firmware
– ATLAS ROD Header• Firmware and registers
– User Header• Registers
– Data payload• Input FPGAs
– Status words• Calculated in Switch from
Input FPGA status bits.
Begin of Fragment (B0F00000 Hex) Start of Header Marker (0xEE1234EE ) Header Size (9 – number of words in the header, excludes control word) Major Format Version No [31:16]=0x0300
Minor Format Version No [15:0] = 0x0
Reserved [31:24] = 0
Sub-detector ID [23:16]
Module ID [15:0]
Run Type Run Number (24 bit) ECRID [31:24] ROD_L1ID (24 bit event no) [23:0] ROD_BCN (12 bit Bunch Crossing Number) Leve1-1 Trigger Type
Detector Specific Event Type Orbit Counter (16 bits)
Sequence Number (12 bits)
Type (4 bit)
Payload – User Header Payload – formatted event data Status Word-1 Status Word-2 No. of Status elements = 2 Number of data elements Status block position – 1(data first) End of Fragment (E0F00000 Hex)
15 Aug 2006 Norman Gee - L1Calo ROD Introduction 24
ROD format – User Header
• Sadly not allowed for in ROD header so must be part of payload
• Up to 4 words in current firmware. Number of words software controlled.
• First word contains no of header words including itself.
• Currently only 1 word is used
– Contains slice number corresponding to L1A.
Header word 1 - Triggered Slice number
CP-CMMs LUT FADC
JEP-CMMs No of Header words including itself
21-Apr-06 Slinkfmt.xls/User Header
4b NH4b
PPMf4b
PPMl4b
CPM4b JEM
4b CMMcp
4b CMMje
4b WordId
15 Aug 2006 Norman Gee - L1Calo ROD Introduction 25
VME Interface Characteristics
• ROD conforms to VME64x.
• Legal VME addresses will always respond to cycles.
– All software uses the same programming model, which knows RW or RO,register length (16 or 32 bits), bitfield positions, etc.
• Programming model contains registers for all functions of all firmware - some not used in some implementations.
• Default firmware is loaded by ACE based on module Geographical address
– no jumpers needed to configure spare modules
• Numbering (of links, FPGAs, etc) starts from 0 in documentation like C++.
• All modules have a read-only module type at their base address. Hardware and firmware revisions are all readable in registers.
15 Aug 2006 Norman Gee - L1Calo ROD Introduction 26
VME Code fragment
ModuleRegister16 { controlPulseReg 1030 PartDependencies { 500 } Attributes { 0x3 CMM.controlPulseReg "Write only“ } State { "" }}
CMM.controlPulseReg { 1 {"Reset Module"} 1 {"Reset TTC"} 1 {"Reset RoI G-Link" } 1 {"Reset DAQ G-Link" } 1 {"Reset CAN Controller" }1 {"Reset I2C Controller" } 1 {"Reset Crate Flash Controller" } 1 {"Reset System Flash Controller" } 1 {"Reset DLL" } 1 {"Clear Parity Errors" } 2 {"Reset FPGAs" Value { 0 {"Nop"} 1 {"Crate FPGA"} 2 {"System FPGA"} 3 {"CrateAndSystem FPGAs"} }} 4 {"Unused" } }
15 Aug 2006 Norman Gee - L1Calo ROD Introduction 27
Handling Errors
• xxx sources of error:
• Errors detected in the upstream module:
– e.g. incoming LVDS or backplane parity errors, saturation, overflow.
– The module must always produce the correct length data, with flags set to indicate errors.
– The ROD will generally report the errors with the affected data. Some common errors are copied to the ROD fragment status words.
• Errors detected in the ROD
– Related to G-Link transmission or protocol failure, or to mismatching events.
– Identified in Input FPGA and reported in substatus word. For RoIs, errors are reported only in the ROD fragment status words.
• No data error or transmission error should cause the ROD to hang or even slow down.
Recommended