Level-1 Calorimeter Trigger Readout Driver FDR/PRR 15 th August 2006 Introduction to the ROD Norman Gee 15-Aug-2006 Norman Gee

Level-1 Calorimeter TriggerReadout Driver FDR/PRR

15th August 2006

Introduction to the ROD

Norman Gee 15-Aug-2006

Norman Gee

15 Aug 2006 Norman Gee - L1Calo ROD Introduction 2

Contents

• Introduction

– Overview; Readout Scheme; Crate Structure; Physical Implementation

• Input and Output Data

– Characteristics of the data; Zero suppression / compression; Example data;

• Requirements summary

• Input data handling

– Readout process; Timeout; Rod Busy; TrigType Timeout; ECReset

• Output Data handling

– RoI Data; Data routing to S-Link; Data Reduction; Replay mode; Monitoring; generation of S-Link packets

• VME Interface, Canbus


DAQRODs To ROS

e/, /hadClusters

(CP)

0.2 x 0.2Jet / ET

(JEP)

0.1 x 0.1

Pre-Processor

(PPr)Analogue

tower sums0.1 x 0.1(~7200)

RoIRODs To RoIB

To CTP

To CTP

8 PPr crates

4 CP crates

2 JEP crates

2 ROD crates

L1Calo Overview

DAQ

RoIs


Processing Module Readout Scheme

FIFO

Glink

Serial Linkto ROD

Dual-port Memory

Shift Registers

40 MHz Processing Module Pipeline

FIFO FIFO

Dual-port MemoryDual-port Memory

L1A

Readout_scheme 03Aug06.cnv

Write

Read

ScrollingMemory


CPU

Preprocessor Crate (1 of 8)

16 PPMsTCM

Disp

RODCrate

(1 of 2)

CPU

CMM

CMM

16 JEMs

Jet/Energy-sum Crate (1 of 2)

RoIsSlice Data

TCM

G-link

S-link

Slice Data to ROS(see text for discussion of maximum

number of S-Links)

CPM Crate (1 of 4)

RoIsSlice Data

CPU

TCM

CMM

CMM

14 CPMs

RODLINK 28Jul03

RoIs toROIB

CPU

PPM

ROD

PPM

ROD

PPM

ROD

PPM

ROD

CP

ROD

CP

ROD

JE

ROD

CProi

ROD

CProi

ROD

JEroi

ROD

TCM

From otherROD Crate

Slice Data

From three otherPreprocessor Crates

From anotherCPM Crate

From Muontrigger

and CTP


Physical Implementation

• 9U * 400 mm Main module

• 9U * 160mm Rear module

DA

Q o

r R

oI d

ata

Rea

d-ou

t dat

from

up

to 1

8m

odul

es

P1

P2

P3

P0

RJ2

RJ3

RJ0

J1

J2

J3

J0

S-Link 0

S-Link 2

S Link 3

S-Link 1

RP2

RP3

GL

GL

GL

GL

GL

GL

GL

GL

GL

GL

GL

GL

GL

GL

GL

GL

GL

GL

VME

S-Link

Swtch

Receive data, Parity check andCompression

+Data Buffering

Receive data, Parity check andCompression

+Data Buffering

RodOverview 25Apr06.vsd


Internal Structure

• 18 G-Links

– handled in groups of 4

• Five Input FPGAs

– data input, compression, formatting

• Switch FPGA

– Data routing to 4 S-Links

• Monitoring FPGA

– Simple event monitoring

• Overall module functions

– VME FPGA, ACE, CPLD, TTC,

– CANbus, power (details in Viraj’s talk)

Input FPGAReceive data, parity check and

compression+

Data BufferingXC2VP20-5FF896

32 + 4

G-Link

G-Link

G-Link

G-Link

96

32 + 4

G-Link

G-Link

G-Link

G-Link

96

32 + 4

G-Link

G-Link

G-Link

G-Link

96

32 + 4

G-Link

G-Link

G-Link

G-Link

96

32 + 4G-Link

G-Link48

Switch,Controller,

etc,

XC2VP30-5FF896

VME64XJ2

VME64XJ3

S-Link 0

S-Link 1

S-Link 2

S-Link 3

TTCrx

TTCrxDecoder

Card

XC2VP20-5FF896

VME64xInterface

Header

TCM

VM

E

CompactFLASH/IBMmicrodrive

ACEController

JTAG

System ACE

JTAG

SwitchFPGA

Spy EventBuffer

CANBus

Clock fanout

46

46

46

46

Pow

erP

CP

ower

PC

Monitor FPGA

Ins RAM(64Kx32)

Ins RAM(64Kx32)

Dual Port RAM(64K x 32)

PCIInterface

(33MHz,64bit)

MPUInterface

XC2VP20-5FG896

BCID NumberBuffer

Event NumberBuffer

Trailer

Trigger TypeBuffer

Status

Data Count

Dual OpticalReceivers









Links 0,1

Links 2,3

Links 4,5

Links 6,7

Links 8,9

Links 10,11

Links 12,13

Links 14,15

Links 16,17

Reset andModule

InitializationCPLD

System Reset

36

61

4 x Rocket I/O32

TTC

44


compression+



compression+



compression+



compression+


RodBlock 04Aug06.vsd


Data Characteristics

• Preprocessor data has two components per channel (64 ch per module):

– Raw FADC output – typically ~five 11-bit values – to cover a calo pulse…

– …and a calibrated ET value in GeV/c (“LUT data”) .

– Most of the time, there is no calo pulse - so FADC output is pedestal + noise• Can reduce data volume by bit-field compression, zero-suppress LUT data.

Cannot discard FADC data (at least at low L1A rate) – needed to check BCID.

• Most other modules also produce two components, but very different from above:

– Integer values in GeV entering or leaving boards;

– Threshold hits – single bits to show if a digital value exceeds a threshold.• Zero suppression expected to be effective - calo cell value rarely exceeds 1 GeV• All information relating to one bunch crossing is read out in a single clock cycle.

• All the data reduction is designed to be lossless.


Example G-Link Data format (JEM)

Jet Elements + Parity

GP

D0


…

GP

D13


GP

D14

Main Jet Counts

GP

EP

JP

D15

Energy Parity Jet Parity

Main & Fwd Jet Counts (Jems 0,7,8,15 only)

GP

EP

JP

D15

Energy Parity Jet Parity

Expanded jet element frame

LVDS Link Down LVDS Serialiser Error

04-Jan-05 Glinkfmt.xls/JEM DAQ

Jet Element data d8 ... d0SELD

2b F0

2b F1

2b F2

2b F3

8b Ex8b Ey8b Et2b M4

2b M5

2b M6

2b M7

2b M0

2b M1

2b M2

2b M3

12b BC Number4*0

11b Frame 3e

11b Frame 3e11b Frame 3h

3b Thr4

8b Ey4*0

11b Frame 2h

12b BC Number


3b Thr5

3b Thr6

3b Thr7

11b Frame 2e

3b Thr1

3b Thr0

3b Thr2

3b Thr3

11b Frame 1e


11b Frame 1h11b Frame 2h 11b Frame 2e

11b Frame 2e

8b Et 8b Ex

11b Frame 2h22*0

11b Frame 3h


Example S-Link Data format

• Sub-Block Header

• Payload created from G-Link data

• Sub-block Trailer (suppressed if there were no errors)

• Note: there is also a “neutral” format. It copies raw data direct from G-Link to S-Link without interpretation. Very important for debugging.

Each JEM Sub-Block consists of the following:Fmt=1

Word Id L1Calo Sub-Block Header - always present

1 1 0 0

Word ID 0 or more trigger tower data words

0 1 0 P P zs

H parity error E parity error

PPm link down

Jet Element pair

Word ID 0 or 1 jet threshold data words (Main jets)

1 0 0 zs

Word ID 0 or 1 jet threshold data words (Main + Fwd jets)

1 0 0 zs

Word ID 0 or 1 energy subsum data words

1 0 1 zs

Word Id L1Calo SubStatus (omitted if no errors)

1 1 0 1GT

GD

DFO

BM

GI

GP

Failing BCN G-Link Timeout

G-Link down

DAQ Fifo Overflow

BCN Mismatch

G-Link Internal Protocol Error

G-Link Serial parity

12-Jan-05 Slinkfmt.xls/ROD JEM Datablock

6b BCN bits 5:0

3b Vers

4b Module

4b Crate

6b Seqno = Slice No

3b Fmt

2b PL

9b Electron data

4b N. Slices

4b NS2 ( 0 )

9b Hadron data

3b Thr4

3b Thr5

3b Thr6

3b Thr0

3b Thr1

3b Thr2

3b Thr3

5b Src Id (20)

5b Src Id (21)

2b JP

5b G-link Pin

2b F3

3b Thr7

8b Ey8b Et5b Src Id

(22)8b Ex

2b M7

2b M0

2b M1

2b M2

2b M3

2b F0

2b F1

2b F2

2*04b

Module4b

Crate6b Seqno

2b M4

2b M5

2b M6


Summary of Requirements

• Receive 16- or 20-bit data from up to 18 G-Links. Check BCNs & parity.• Optionally discard some timeslices from different sections of source

modules.• Compress or zero suppress data from each channel, as required.• Write formatted data to the FIFO buffers (internal to the FPGAs)• Receive CLOCK, BCR, ECR, event number and event type from TTC.• Select data to be transferred to the ROS based on trigger type.• Build and write ATLAS-standard event fragments to up to four S-Links.• Spy on events (FPGA or VME), optionally select according to event type.• Generate BUSY signal to the Central Trigger Processor.• Process complete events at up to 100 KHz sustained rate.• For RoI data only, limit the number of RoIs to a programmable maximum.• Support CANBus board temperature and voltage monitoring.


Description of readout process

• At Source Module

– L1A arrives, n (e.g. 5) timeslices copied into readout FIFOS in G-Link data format.

– DAV asserted. Data for n slices sent, with parity after each. DAV removed

• At the ROD

– L1A arrives, BCN stored in FIFOs in Input & Switch FPGAs

– Input FPGA: G-Link data received, reformatted, zero-suppressed, stored in data FIFO.

• Checks on G-Link status, Longitudinal parity, BCN -> error bits in substatus word

– Switch FPGA: receives event type.

– Switch FPGA: waits for all Input FPGAs to complete; creates S-Link header; copies payload to S-Link; creates S-Link trailer;


Input data timeout

• Happens if a source module fails to provide expected data.

• Input timeout is detected by Switch FPGA:

– Start a timer when the first Input FPGA has processed data available.

– Declare Input timeout if timer expires before all expected Inpout FPGA channels have processed data available.

• When building the S-Link packet, timed-out Input channels are instructed to provide a sub-header and a sub-status word with timeout flag set.

– Late data is not used even if available by then.

• If the missing data does eventually arrive (unlikely), it will be put into the following event

– but will have wrong Bunch Number.

• Any surplus data is erased by the ECR mechanism.


Generation of ROD- Busy

• Four S-Links - each can assert LFF (“Xoff”) if destination is too slow.

• When LFF, Switch stops working. Input FPGA Data FIFOs start to fill up.

• When depth limit is reached, ROD-BUSY is generated.

– Programmable depth limits on three FIFOs – event data, event management, BCN FIFOs.

– Must be set to handle worst case – 8 events separated by 5 bc each.

• We have two sets of counters:

– one set to monitor the fraction of time each S-Link is asserting LFF.

– one set as an 8-bin histogram of Input FPGA data FIFO depth.


Trigger-type timeout

• Trigger type is sent by TTC broadcast so has no guaranteed arrival time

– depends on other TTC channel B traffic.

• Trigger type is needed to populate ROD header.

• If ROD waits indefinitely, Input FPGA FIFOs will fill up and ROD BUSY will be generated.

– If trigger type doesn’t ever arrive, ROD will block.

• Trigger-type timeout starts once processed data for an event is ready in the Input FPGAs.

• After the timeout, a default value will be used.

• If trigger type arrives very late, it is cleared by ECR


Event Counter Reset

• Sent by CTP at about 1 Hz. Ignored by TTCrx ASIC, but used in ROD to clear ROD-internal 24-bit L1_ID and increment 8-bit ECR Counter.

• Both together form 32-bit ROD_L1ID field in S-Link header.

• Also usedin L1Calo RODs to clear any partial events.

ECR Deadtime

L1AWorst Case

S-Link Readout

ECR

Buffer Reset

ECRTiming 30Jul03


RoI Data

• Collected from source modules just like DAQ data

– One timeslice only.

• Sent to separate “RoI-RODS” on separate links. Zero-suppressed as DAQ data.

• Collected by Switch FPGA and counted as sent to RoIB over S-link

• Programmable limit to number of RoIs sent.

– When this limit is reached, no more RoI payload is sent. Status bit is set.

– This guarantees that RoIB is not overwhelmed by wild events in ATLAS

• Normally only 1-2 RoIs expected per event.

– Calibration runs will produce many more – not used in Level-2.


Data Routing to S-links – DAQ Data

• Data from each Input FPGA (4 G-Links) is concatenated into a separate ROD fragment. Data from the last two input FPGAs (nos 3 & 4) are combined into one fragment

– G-links 0 - 3 -> S-link 0

– G-Links 4 – 7 -> S-link 1

– G-Links 8-11 -> S-Link 2

– G-Links 12-17 -> S-Link 3

• All data from Input FPGAs 3 & 4 (4 or 6 G-Links) is concatenated into S-Link 3.

S-Link 0

S-Link 1

S-Link 2

1

2

3

4

5

6

7

8

9

10

11

12

13

14

DAQ

DAQ

15

16FPGA

+Data

Buffers

S-Link 3DAQ

0-3

4-7

8-11

12-15

ROD - Preprocessor, CP, JE Slice Data S-Link RTM

SwitchSwitch

Switch

FPGA+

DataBuffers

FPGA+

DataBuffers

FPGA+

DataBuffers

FPGA+

DataBuffers

17

S-Link DAQ 25Apr06

DAQ

0

16-17


Data Routing to S-links – RoI Data

• All data from 18 G-Links is concatenated into a single ROD fragment

• The fragment is duplicated to Slinks 0 and 2.

– S-Link 2 has a logic analyser header.

S-Link 0

S-Link 1

S-Link 2

17

0

FPGA+

DataBuffers

DAQ

RoI

FPGA+

DataBuffers

S-Link 3

0-3

4-7

8-11

12-15

ROD -RoI Data S-Link RTM

Switch

Switch

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

FPGA+

DataBuffers

FPGA+

DataBuffers

FPGA+

DataBuffers

Slink_RoI_26Apr06

16-17


Data Reduction

• Most data (apart from PPM) is subject to zero-suppression. S-Link words are not written if the data values and error bits are all zero.

– Typically expect only a few percent of non-zero trigger towers.

– For 5% occupancy of trigger towers, achieve factor 5-10 reduction

• Addressing information is added to identify remaining data:

– in ROD header (identifies DAQ and RoI, plus crate family)

– in Sub-block header (identifies source crate, type of module, slice number)

– in S-Link words (identify data source within module)

• For the PPM, zero-suppression is not appropriate, so a variable-length bit field approach is used.

– Based on subtracting FADC samples to give small numbers, which can be packed in short bit-fields. Expect a factor 2 data reduction.

– Prototype scheme is documented and checked. Will be tuned.


Replay Mode

• Replay mode runs from the Input FPGAs.

– Disable all G-Links;

– Preload FIFOs with required data and put lengths in event management FIFOs.

– Generate L1As internally (software) or using TTC system.

– Transmitted data is recycled back into the FIFOs so one or several events can be played continuously

• BCN, Event number and Trigger Type are obtained from fixed registers, not TTC.


Event Sampling and Monitoring

• Event sampling buffers are provided in the Switch FPGA for each S-Link. Sampled events may be accessed from VME or copied to the Monitoring FPGA.

• Monitoring FPGA code is not yet written. Data will be available for monitoring by code in FPGA-internal PC cores or to PCI daughterboard.

• Events are eligible for sampling if they satisfy an event-type mask and a prescale factor.

– All S-Links are sampled (or not) on the same event.

• Two sampling modes:

– Random: events sampled whenever a buffer is available;

– 100%: All events sampled. Software LFF is asserted if all buffers are full. This mode will assert ROD-BUSY if the trigger rate is too high.

• A mechanism is provided to sample the same events in all RODs


Generation of S-Link fragments

• 5 components:

– BOF and EOF control words• Fixed in Firmware

– ATLAS ROD Header• Firmware and registers

– User Header• Registers

– Data payload• Input FPGAs

– Status words• Calculated in Switch from

Input FPGA status bits.

Begin of Fragment (B0F00000 Hex) Start of Header Marker (0xEE1234EE ) Header Size (9 – number of words in the header, excludes control word) Major Format Version No [31:16]=0x0300

Minor Format Version No [15:0] = 0x0

Reserved [31:24] = 0

Sub-detector ID [23:16]

Module ID [15:0]

Run Type Run Number (24 bit) ECRID [31:24] ROD_L1ID (24 bit event no) [23:0] ROD_BCN (12 bit Bunch Crossing Number) Leve1-1 Trigger Type

Detector Specific Event Type Orbit Counter (16 bits)

Sequence Number (12 bits)

Type (4 bit)

Payload – User Header Payload – formatted event data Status Word-1 Status Word-2 No. of Status elements = 2 Number of data elements Status block position – 1(data first) End of Fragment (E0F00000 Hex)


ROD format – User Header

• Sadly not allowed for in ROD header so must be part of payload

• Up to 4 words in current firmware. Number of words software controlled.

• First word contains no of header words including itself.

• Currently only 1 word is used

– Contains slice number corresponding to L1A.

Header word 1 - Triggered Slice number

CP-CMMs LUT FADC

JEP-CMMs No of Header words including itself

21-Apr-06 Slinkfmt.xls/User Header

4b NH4b

PPMf4b

PPMl4b

CPM4b JEM

4b CMMcp

4b CMMje

4b WordId


VME Interface Characteristics

• ROD conforms to VME64x.

• Legal VME addresses will always respond to cycles.

– All software uses the same programming model, which knows RW or RO,register length (16 or 32 bits), bitfield positions, etc.

• Programming model contains registers for all functions of all firmware - some not used in some implementations.

• Default firmware is loaded by ACE based on module Geographical address

– no jumpers needed to configure spare modules

• Numbering (of links, FPGAs, etc) starts from 0 in documentation like C++.

• All modules have a read-only module type at their base address. Hardware and firmware revisions are all readable in registers.


VME Code fragment

ModuleRegister16 { controlPulseReg 1030 PartDependencies { 500 } Attributes { 0x3 CMM.controlPulseReg "Write only“ } State { "" }}

CMM.controlPulseReg { 1 {"Reset Module"} 1 {"Reset TTC"} 1 {"Reset RoI G-Link" } 1 {"Reset DAQ G-Link" } 1 {"Reset CAN Controller" }1 {"Reset I2C Controller" } 1 {"Reset Crate Flash Controller" } 1 {"Reset System Flash Controller" } 1 {"Reset DLL" } 1 {"Clear Parity Errors" } 2 {"Reset FPGAs" Value { 0 {"Nop"} 1 {"Crate FPGA"} 2 {"System FPGA"} 3 {"CrateAndSystem FPGAs"} }} 4 {"Unused" } }


Handling Errors

• xxx sources of error:

• Errors detected in the upstream module:

– e.g. incoming LVDS or backplane parity errors, saturation, overflow.

– The module must always produce the correct length data, with flags set to indicate errors.

– The ROD will generally report the errors with the affected data. Some common errors are copied to the ROD fragment status words.

• Errors detected in the ROD

– Related to G-Link transmission or protocol failure, or to mismatching events.

– Identified in Input FPGA and reported in substatus word. For RoIs, errors are reported only in the ROD fragment status words.

• No data error or transmission error should cause the ROD to hang or even slow down.

Documents

Level-1 Calorimeter Trigger Readout Driver FDR/PRR 15 th August 2006 Introduction to the ROD Norman Gee 15-Aug-2006 Norman Gee