Marcel Stanitzki Yale UniversitySilicon Workshop 2006 Santa Barbara
FIB & FFOBrothers in crime
Marcel StanitzkiYale University
Marcel Stanitzki Yale UniversitySilicon Workshop 2006 Santa Barbara
The Big Picture
SVX DAQ is big,
but not really complicated !
Focus of this talk
Marcel Stanitzki Yale UniversitySilicon Workshop 2006 Santa Barbara
Lovely Acronyms
● FFO: FIB FanOut● FIB: Fiber Interface Board● FTM: FIB Transition Module● VME bus: VERSAModule Eurocard bus
Marcel Stanitzki Yale UniversitySilicon Workshop 2006 Santa Barbara
The FFO
● Basically a Signal/Command Distributor (1 SRC -> n FIBS)
● Necessary as a local Clock source for a FIB crate (in absence of an SRC)
● The FFO has a lot of diagnostics features to report back to the SRC or via VME
FFO
Commands/Clock from SRC
ClocksCommands
ClocksCommands
J3 Backplane
Marcel Stanitzki Yale UniversitySilicon Workshop 2006 Santa Barbara
The FFO (Rev B)
● 9 U VME Board● 1 GLINK Receiver & 1 RS-485
port● Custom J3 Backplane (CDF FIB)● Firmware cannot be upgraded via
VME● Also used for the Resonance
Protection System (See Ghostbuster Talk)
Marcel Stanitzki Yale UniversitySilicon Workshop 2006 Santa Barbara
The FFO in SVXDAQ
Diagnostics and FIFO control (You'll never use those)
FFO Operation Mode
Controls for Remote Reset and Resonance
protection
GLINK Status
State MachineInfo
Board RevisionFirmware version
Marcel Stanitzki Yale UniversitySilicon Workshop 2006 Santa Barbara
FFO failures● Very stable board, very few problems so far● Main problem with FFO's is the GLINK Sync with the SRC
– 1 Board was swapped in 2005 due to various GLINK sync losses– Only swap so far (as I recall)
● Some Problems with the READOUT RETURN line addition in early 2004 (fixed in firmware)
● Discovery of a logic race between SRC/FFO in Autumn 2004– Needs very specific conditions in order for it to happen– If it happens, it is spectacular, entire detector might trip.– Protected by software
Marcel Stanitzki Yale UniversitySilicon Workshop 2006 Santa Barbara
The FIB - what does it do ?
● One FIB board controls 2 wedges (= 10 HDI's)● Receives commands from SRC, translates it into SVX3 D
command sequences and then send them to the HDIs● Provides FrontEnd (132 ns) and BackEnd (19 ns) Clock to
the Chips● Receives Data from the DOIMs, does some basic
processing and sends it to the VRB via GLINKS
Marcel Stanitzki Yale UniversitySilicon Workshop 2006 Santa Barbara
The real thing
● 9U VME Board● 17 FPGAs (Altera)● Custom J3 Backplane
(CDF-FIB)● 4 GLINK Transceivers● Firmware can be
updated via SVXDAQ
Marcel Stanitzki Yale UniversitySilicon Workshop 2006 Santa Barbara
FIB box diagram
VME Int erface
J3
GLINK Controller
GLINK 0
GLINK 1
GLINK 2
GLINK 3
FIB Command Sequencer
& Chip Clock Generator
VME
Con tro l
01234
01234
SRC Commands & Clock via FFO
Pipeline Controller
Chip SequencesFE/BE clockData Wedge A & B
Data to VRB
FIB Configuration via VME
e.g. SVXDAQ or Runcontrol
Marcel Stanitzki Yale UniversitySilicon Workshop 2006 Santa Barbara
FIB Command Sequencer
● Commands are queued in Command FIFO
● Simple message turns into complicated control sequence
● FIFO can be stopped, skipped, current sequence can be interrupted – Timing matters!
● Static RAM and Sequencer can be overwritten (via VME)
Marcel Stanitzki Yale UniversitySilicon Workshop 2006 Santa Barbara
FIB PipelineData
● A FIB has 10 Pipelines● Pipeline Processor inserts FIB headers/End of record● Can perform Pedestal Subtraction (Calc/Ped RAM). We
do not use this● UnGray Data (SVX3D writes data in Gray code)
Marcel Stanitzki Yale UniversitySilicon Workshop 2006 Santa Barbara
FIB Data Stream
The FIB writes “The self describing data” (S.NAHN)
e0002cfc 8319003e 013e023a 033f043d 0b3b0c29 0d3a0e39 133a1438 153a163a 25392628 2738283b 2a392b38 2c3a8219 a1194d39 4e394f39 553b5628 573ea019 7f01c1c1
● FIB Header: 2 bytes ● Bunch Crossing (BX)/ BackEnd state(BE) : 2 bytes
– BX/BE from SRC, central for all 10 HDI's● Chip ID/CAPID: 2 bytes per Chip● End of Record (EOR) :2 bytes
For 1 HDI :
Marcel Stanitzki Yale UniversitySilicon Workshop 2006 Santa Barbara
The FIB Header
● W:– e or SVX f for ISL
– f for L00
● X:– bulkhead 0-5
(SVX/ISL)
– 8 for L00 West, a for East
● Y:– Wedge 0-b for SVX
– Wedge 0-5 for ISL/L00
● Z– Layer 0-4 (SVX/ISL)
– Layer 0-3 (L00)
Format is WXYZ
WXY can be set freely via VME (just a name)
Marcel Stanitzki Yale UniversitySilicon Workshop 2006 Santa Barbara
FIB EOR
● C1C1 : Data stream ok● C2C2 :
– No data – Data stream corrupt (e.g not last chip last channel)– Max number of BackEnd clocks exceeded– 00F3 8 identical bytes in a row
● C4C4 : Fill Characters● C8C8 :
– Event abort (e.g. SRC L2R)
FIB terminates data stream from a HDI with CXCX
Marcel Stanitzki Yale UniversitySilicon Workshop 2006 Santa Barbara
GLINK
● Commercial Protocol (originated by HP)● Serial Protocol, 1.5 Gigabit/s● Transmitted via Fibers (Finisar Drivers 850 nm
Wavelength)● max 20 Bit per GLINK● CDF's funny encoding
– 4 GLINKs 80 Bits = 10 HDIs with 8 bit each– 1 GLINK e.g. HDI 0/1 + 4 Bits of HDI 4
● GLINK needs synchronisation
Marcel Stanitzki Yale UniversitySilicon Workshop 2006 Santa Barbara
What is D0 ?
● GLINK Data stream must to be continuous
● DOIM Data stream is not● For every gap, 0xD0 are
inserted (Fill bytes)● This avoids Data UnderRun● The VRB removes them (see
Tuula's Talk)
CHIP Data Data
DOIM Data
GLINK Data
D0D0 D0D0
Marcel Stanitzki Yale UniversitySilicon Workshop 2006 Santa Barbara
The Blinking Lights ...VME Bus access
Voltages (3 LEDs, must be on)
RESET
Voltages (all on) VME Bus access
FFO Mode (RUN) Clock
(SRC GLINK(LOCK)FIFOS
(OFF) RESET
FFO Return cableto SRC
GLINK INPUT from SRC
GLIN
K O
UT
to VR
BThe Blinking lights tell it all !
(Yale Mantra)
Marcel Stanitzki Yale UniversitySilicon Workshop 2006 Santa Barbara
The FIB in SVXDAQ
Set FIB header(WXY)
Channel enables
GLINK ControlsDigitize/Readout
Clocks
Set Last CHIP ID
HDI <-> PipeMatching
Clock controls
MicroSequencer Controls (Don't touch)
PCB VersionECO LevelSerial No.
Marcel Stanitzki Yale UniversitySilicon Workshop 2006 Santa Barbara
The FTM
● Connects the HDI's to the FIB
● Uses the CDF FIB J3 Backplane (non Bus part)
● 2 Control Cables (A/B) for 2 Wedges using LVDS
● 10 Optical Receivers (aka RX) for the DOIMs from 10 Ladders (see Sasha's Talk)
Wedge B
Wedge A
Wedge B
Wedge A
Data
CommandsClock
FTMFIB
Marcel Stanitzki Yale UniversitySilicon Workshop 2006 Santa Barbara
BackEnd Kludges
● Dead OBDV in one channel● BackEnd Kludge Fix
– Take clock from Command line– Add delay (adjustable, infamous
potentiometer)– Use this to clock in the incoming data
● Requires special Pipe Firmware● And Requires “Last Chip Override” set in
the Chip Editor
Marcel Stanitzki Yale UniversitySilicon Workshop 2006 Santa Barbara
FIB failure modes
● Pipeline Failures – Bit errors in one HDI
● Command Clock Failure– Two Wedges affected (Trips, corrupted data, C2C2)
● C2C2 for one ladder– Pipeline dead– May be RX problem, check FTM
● Stuck VME Bus– Fuses blown, swap the board (Usually one FPGA died)
Marcel Stanitzki Yale UniversitySilicon Workshop 2006 Santa Barbara
A FIB VME Crate
● 1 VME Crate CPU● Up to 12 FIB's● 1 FFO● FIB J3 from Slot
8-20● FIB's only work
there !● J3 is fragile,
Be careful !
A FIB Crate contains :1 2 3 4 5 6 7 8 910 11 12 13 14 15 16 17 18 19 20 21
MVME 2401
FIB
FIB
FIB
FIB
FIB
FIB
FIB
FIB
FIB
FIB
FIB
FIB
FF0
FIB J3
VME SLOT
VME Crate Controller
9U
Marcel Stanitzki Yale UniversitySilicon Workshop 2006 Santa Barbara
Summary
● Important to know what the FIB and FFO can do !– Helps Diagnosing problems– Stop putting all the blame on the DAQ
● The manuals cover quite an amount of detail● Many “odd” things have a not always obvious reason for
being that way● If you don't know, you should ask people ...
Marcel Stanitzki Yale UniversitySilicon Workshop 2006 Santa Barbara
Backup Slides
Marcel Stanitzki Yale UniversitySilicon Workshop 2006 Santa Barbara
FFO Race● In order for this thing to
happen, you'll need to have several things coming together– The SRC processes non-fatal
errors
– The FFO return cables (RS-485) are connected
– The FFO has to report an error on these error lines
– for the crates to reboot, you need to have system reset enabled on the FFO.
SRC
FFOError reporting
Clear the error
SRC is too fast for FFO
Marcel Stanitzki Yale UniversitySilicon Workshop 2006 Santa Barbara
FFO Race Fix
● Make sure SRC Process Not Fatal Errors is Disabled (happens on each CONFIG)
● Error line 0-2 on all FFO's are disabled (3 is used for resonance protection and not affected)
Marcel Stanitzki Yale UniversitySilicon Workshop 2006 Santa Barbara
FFO GLINK Blues
● If a FFO looses the Glink sync, an entire FIB crate is affected
● Resyncing the GLINK– Do a SRC Reset Glink -> Sync
Glink– do a VME reset of the FIB
crate– Powercycle the crate
Marcel Stanitzki Yale UniversitySilicon Workshop 2006 Santa Barbara
Gray Code
● Alternative Binary Coding System● Only 1 Bit changes at
incrementing a number● Originally invented to solve
electromechanical problems● For Chips
– Less Power consumption– More Robust ADC
Decimal Binary Gray0 000 0001 001 0012 010 0113 011 0104 100 1105 101 1116 110 1017 111 100
Marcel Stanitzki Yale UniversitySilicon Workshop 2006 Santa Barbara
Timing● After L1A, FIB performs DIGITIZE (fixed
time/event)– 2.26 µs setup/cleanup + 4.57 µs ADC ramp
● then does READOUT (occupancy driven)– 1.3 µs setup/cleanup + 19 ns/Byte
Marcel Stanitzki Yale UniversitySilicon Workshop 2006 Santa Barbara
Command Cable Signals
● FE_CLK FrontEnd Clock (132 ns)● BE_CLK BackEnd Clock (19 ns)● C_CLK Command Clock (38 ns)● C Command 6 bits● C_DATA Command DATA● L1A Level 1 Accept● PIPE_RD2
Marcel Stanitzki Yale UniversitySilicon Workshop 2006 Santa Barbara
Downloading FIB Firmware
● 2 kinds of “Firmware“– FIB Sequences ( MS RAM menu)– FIB Firmware (JAM menu)
● Checking Firmware Version– Sequences : Checksum– JAM : Use Verify
Marcel Stanitzki Yale UniversitySilicon Workshop 2006 Santa Barbara
FIB special value
● For Data, the values 0xEF to FF are reserved● Most prominent is 0xF3
– Data from Chip is 0x00 (after DPS)
– FIB replaces this with 0xF3● Other error codes are 0xF1,0xF2 ● See CDF/DOC/CDF/PUBLIC/4152
Marcel Stanitzki Yale UniversitySilicon Workshop 2006 Santa Barbara
Super/Duper FIB
● Replacing chips on the input fifo to increase the tolerance of varying duty cycles on OBDV coming from the SVXII chips
● Add the "ISL Mod" which increases the width of FE Clock HIGH, adding some breathing space for edges which must fall inside a FE Clock
● This upgrades the FIB to ECO level 5
Marcel Stanitzki Yale UniversitySilicon Workshop 2006 Santa Barbara
SRC->FIB commands to know
● 0x00 No operation● 0x06 Digitize● 0x05 Readout● 0x0C KeepAlive● 0x02 Preamp Reset Start (XQT=1)● 0x04 Preamp Reset Stop (XQT=1)