A READOUT SYSTEM-ON-CHIP FOR A CUBIC KILOMETER SUBMARINE

NEUTRINO TELESCOPE

S.Anvar, V.Gautard, H.Le Provost, F.Louis, K.Menager, Y.Moudden, B.Vallage, E.Zonca, on behalf of the KM3NeT consortium1

IRFU/SEDI-CEA SaclayF-91191 Gif-sur-Yvette, France

1 Supported by the European Commission through FP6 and FP7

TWEPP-11, Vienna, Austria, 26-30 September 2011

The KM3Net Detector

Digital Optical Module (DOM)

31 PMTs/DOM

12800 DOMs

320 lines

http://www.km3net.org

Technical Design Report (ISBN 978-90-6488-033-9) Conceptual Design Report (ISBN 978-90-6488-031-5)

The DOM processor board

HV PMT base

Signal collection board

Processor board & Softwaredesigned at CEA-IRFU

Power conversion boardHeat conductor

PMT

DOM designed at NIKHEF institute (Amsterdam, Netherlands)

RTOS

DDR2Memory(64 MB)

FlashMemory(32 MB)

FPGABitstream

/Processor

Boot

The KM3Net prototype Offshore Processor

Board

PPC440 processor

SlowControl

SlowControl

Task

Slow-Controlfor the Storey (I2C, SPI)

1Gb/s Ethernet LinkTo shore station

DataTask

Data

ReadoutLogic

Readout System On Chip (RSOC) / VIRTEX-5

SC ProtocolLogic

Clock/CommandExtraction

PMTs

x 31 small PMTs

31x

1 GHzTDC

31 TOT Data

Firmware TDC Development

1 GHzSampling

PLL500 MHz

/500 MHz

125 MHz

RecoveredETHERNET

Clock@125MHz

XILINX VIRTEX-5 ISERDES

Readout Logic@125 MHz

‘1’ Detection&

Coarse Time Stamping

1 1

1

0 0

0

0 0

0 0 1 1 1 0 0

TOT LVDS PMTasynchronous input

8-bit

0

Packetized and Sent over ETHERNET

TDC #16-bitCoarse Time Stamp #24-bitEvent type/TDC number

[Originally designed by A.Zwart (NIKHEF) / small PMTs test bench for ALTERA]

Clock distribution

Transmit Reference Clock @62.5 MHz

STANDARD1000BASE-XETHERNET

Phase and Latency

Measurements

Command (CMD) insertion

Custom Logic

On Shore VIRTEX-6 Board

IDLE CMD DATA

Command(CMD) extraction

Custom Logic

Off Shore VIRTEX-5 RSOC

1.25 Gbps

PPC440 Bus

Recovered Clock @62.5 MHz / Bit slice analysis (known latency)

CMD

IDLE

DATA

Event Time Stamping

Embedded62.5 MHz

Transmit/Receive clock skew

0 2000 4000 6000 8000 10000 12000 14000 160004.4

4.5

4.6

4.7

4.8

4.9

5

5.1

5.2

5.3

Time since cold start in seconds

Tx/Rx clock skew measured on shore with DMTD and oscilloscope (ns)

DMTD (Dual Mixer Time Difference)

Scope

Store & Forward Acquisition model

31 TDC

&

RoutingMatrix

Dynamic Memory

AcousticEvent

Format(128 bytes)

TDC event

format(6 bytes)

CircularBuffer

FIFO 0

PMT 0-31

PPC440 processor@ 300 MHz

1 Gb/s Ethernet Port

134 ms Data

Time Slice To Computer 1

Time Slice To computer 2

Interrupt@134 ms(Typical)

Slow control (Configuration, Readout on requests)

134 ms Data

Detector Clock(ETHERNET)

Slow ControlObjects

FIFO 5

DAT

A RO

UTE

R

ReadoutLogic

Acoustic Data

FIFO 6

FIFO 0

CircularBuffer FIFO 6

134 ms

134 ms

Time Slice BuildingIntrinsic Parallelism

Performance Parallelism

OffshoreNodes

OnshoreFarm

Multi-gigabit Standard Ethernet Switching

FarmComputing

Clock Distribution/Commands Insertion Detector Clock(Start) Commands

• Track reconstruction• Data routing

EmbeddedComputing

Time Slice 2

Time Slice 1Time Slice 0

Start

NetworkEmbedded

TargetSC Server

RunControl

Data routers / writer

ElectronicsControl & Config (SC & Elec. RC)

EmbeddedTarget

SC ServerCommunication

middleware (ICE)

Communicationmiddleware (ICE)

Electronics control & config

Data acquisition setup

Reference Clock @62.5 MHz

STANDARD1000BASE-XETHERNET

Start Command

On Shore VIRTEX-6

Board• Run Control• Target Configuration• Data acquisition• vxWorks RTOS boot

server

Host PC

Synchronous1000BASE-XETHERNET

RSOC VIRTEX5FX-70• vxWorks RTOS• 1 TDC Channel• 1 dedicated TDC ASIC

channel• PPC440@300 MHz

Bus@75 MHz

Pulsegenerator

Acquisition results

8 ns2 ns 2 ns

Standalone measured TCP/IP throughput PPC440@400 MHz / Bus@100 MHz /WindRiver Zero Copy buffer/Jumbo frames : 988 Mb/s

Current acquisition setup (Source limitation): 60 Mb/s

Pulse @ f=10 kHz

ns ns

Scope measurementMean : 100,10667 usRMS : 1,75 ns

Power : 7 W

CONCLUSION

Common Readout system functions integrated in a single component (RSOC):Event Time stamping @ 1 GHzClock and command distributionSlow-control and data acquisition performed in a RTOS

multi-tasking embedded system

RSOC is a node designed to be plugged in a complete Data acquisition SystemServer/Client topology (ICE)Scalable system