Timepix3 Readout
By: Bas van der Heijden & Martin van Beuzekom
3/20/2013 1Nikhef
Nikhef 2
MPX3 and TPX3 readout -> SPIDR• Speedy PIxel Detector Readout• Development on a Xilinx evaluation board
– design a custom (small form factor) FPGA board at a later time
• Aim to read out 1 Timepix3 at full (80 Mhits/s) speed, or multiple Timepix3 chips at lower speed (less links per TPX3)
• Readout bandwidth at least 5 Gbits/s -> use 10 Gigabit ethernet– currently using 1 Gigabit ethernet– 10 Gbe VHDL code development started
• MPX3.1 and RX readout up and running (low DAQ speed)• Re-use ‘back-end’ code for TPX3 readout
3/20/2013
Nikhef 33/20/2013
Xilinx ML605 development board
Medipix3 chipboard
FMC connector
Timepix3 chipboard
TPX3
69mm
150m
m
lemo00Te
st fixtu
re 11
xPower supply1.5V
HV for sensor/ gridNo routing
3/20/2013 4By: Bas van der Heijden Nikhef
test fixture for diff. probe
Timepix3 chipboard• FMC (high pincount 8gbt’s)• Mux for selecting between gbt and regular IO pins on fpga (PI2PCIE2412ZHE)• ADC & DAC for timepix I2C• Lemo’s (trig/clk/busy/?)• VDD(1.5V) VDDA(1.5V) VDDPLL(1.5) (from one DC/DC? CERN sm01c)• VDDA(3.3 efuse from FMC)• U&I monitor (ina219)• CERN rad-hard powersupply’s?• Flash for storing pixel mask (mp25p32 32Mb)• Simple bias supply for Si sensor? (max668) (12-100V)• Extra pad and connector/pad for HV bias (1000V) (thick sensor/grid+cathode)• Probing fixtures for 8x data_out/data_in/clk_in/enable_in• No routing behind chip (allows removing/thinning of PCB for beamtest)• Chip cooling?
3/20/2013 5By: Bas van der Heijden Nikhef
Nikhef 6ToA 14+4b ToT 10bAddress 16bCrtl 4b Coarse Timestamp20b
Data8b
CLK64MHz
64MHz(recovered clock)
K-char WRen
Control data FIFO
125MHz 156MHz
10GBMAC
10GBPCS
TXBUF-FER
RXBUF-FER
Data 68b
CLK10MHz
Data 68b
Empty
RdReq
Almostfull
Throttle control
IP MU
X
CPU
TimeStamp14+20bits2^34*25ns ≈ 7min
Unique timestamps@40MHz
64b
Port 1
Port n
TPXCLK
TPXRST
XGM
II
10GBase X
Fast data transmission path
sop
sop
Data 68b
8x
-Remove control header-Send packet on timeout or max size
MAX data rate is 5.12Gbit(80MHz*64bit@8 lanes)
Pixel word 68b
timestamp20b
eop
eop
SerDes
640Mbit
FIFO
68b
(dep
th=1
6)
Pixel word collector
48b + 20b timestam
p
Data 68b
Empty
RdReq
Empty
RdReq
WO
RD M
UX
FIFO68b
Depth=40978x36Kb EBR
Data 64b
Cont
rol w
ord
DEM
UX
Empty
RdReq
sop
eop
Pixel data to Ethernet
packet State
Machine
Data64b
Data64b
3/20/2013
Almost full
Nikhef 7
Fast data path summary
• Add 20 bit timestamp to 48 bit pixel packets from TPX3• data of up to 8 serial links merged into one fifo (round robin)• strip off 4 bit ctrl header in ethernet packet builder• use 4 bit ctrl header to route data (to GBE / CPU)• GBE packet transmitted when either max. length reached or
when time-out (configurable)– max. collection time given by 14+20 timestamp @40 MHz -> 7 minutes
3/20/2013
125MHz 156MHz
10GBMAC
10GBPCS
TXBUF-FER
RXBUF-FER
IP MU
X
CPU
64b
64b
Port 1
CPU Port
XGM
II
10GBase X
control data path40/80MHz
RXStatemachine
Packet decoderData outFront end
Enable
Data
ClkIn
Control signals
T0 sync
Shutter
TPulse
MU
X
3/20/2013 8Nikhef
Throttle controlVeto
ext_T0 & shutter
Nikhef 9
control path summary• Configuration / control via soft-core CPU (Leon)• Configuration directly from GBE also possible via Rx statemachine• Rx statemachine also for throttling (pause frames)• The Soft-CPU + firmware take care of the HW specific details
• Testpulse generator• Shutter control
3/20/2013
Throttle structure
DAQapplication
Network card10G
Base X
MAC+PCSTX State-MachineFIFOTimepix3
DAQ busy packet(continue after …)
Pause frame
FIFO almost full
FPGA PC
Shutter
Other detectors
Throttle controlFrontEnd disable
FE disable from other detector
3/20/2013 10Nikhef
Nikhef 11
Hardware status + plan for next 3 months
• 2 hardware engineers full time + support for PCB design• MPX3+RX readout working
– 1 lane to MPX3, 1 Gbe to PC– not yet all modes are implemented/tested– further VHDL development is temporarily stopped; focus on TPX3 specific parts
• Aim to have ready before TPX3 returns from fab:– Chipboard– 10 GBE (fallback to 1 GBE possible)– VHDL statemachine, interface to TPX3– Pixel packet receiver and packet builder for 10 GBE
• Later this year: – features and testing max. data throughput– custom FPGA board– tileable chipboard for MPX3 and TPX3 (2 versions)
3/20/2013
Nikhef 12
Software status and plans • Simultaneous code development for soft-CPU in FPGA and driver library
at PC side• Library (“API”) for MPX3+RX functions almost complete, being
tested/debugged at the moment• Low level TPX3 calls will be added the coming months• Testprograms (“scripts”) are simply a collection of calls from the library
– Calls like: “SetDACs”, “openShutter”, “ReadFrame” etc.– written in C/C++, compile before running (MS Visual Studio Express 2010, free
version)
• Pixel data written to disk and /or on primitive display – decoder for offline display (Root) will be provided
• Additional layer of code will be added to adapt to Pixelman API • High speed DAQ program will follow later this year
3/20/2013
Nikhef 13
Software tools
• (V)HDL graphical entry tool: Ease from HDLworks• Simulation: Questasim latest version• Synthesis: Xilinx ISE version 14.x• Use of SVN
• Software development: MS Visual Studio Express 2010– free version
•
3/20/2013
Nikhef 14
We would like to have/know:
• Final pinout / dimensions of TPX3 for chipboard• Up to date manual (working document?)• Simulation model of TPX3
Other• S-LVDS issue? converter needed?
3/20/2013