IRFU

The ANTARES Data Acquisition System

S. Anvar, F. Druillole, H. Le Provost, F. Louis, B. Vallage (CEA)

ACTAR Workshop, 2008 June 10

IRFU

The ANTARES detector

http://antares.in2p3.fr

IRFU

Acquisitionnodes

LineControl

Junctionbox

Electromechanicalcable

Electro-opticalcable

Slow-controlDataClockEnergy

Processing nodes

OffshoreOnshore

12 detection lines (400 m)12 detection lines (400 m)

300 acquisition nodes (25 / line)300 acquisition nodes (25 / line)

900 photomultipliers (3 / node)900 photomultipliers (3 / node)

1800 data sources @ 20 Mb/s max1800 data sources @ 20 Mb/s max

System spread over 30000000 m3 System spread over 30000000 m3 @2500 m depth@2500 m depth

Onshore Farm of 80 processing Onshore Farm of 80 processing nodesnodes

The "0.1 km2«  project

photomultiplier

IRFU

ANTARES Sector Line

IRFU

ANTARES Line deployment (1)

IRFU

ANTARES Line deployment (2)

IRFU

Photomultipliers counting rates

bioluminescence burst (April 2003)*

seconds

kHz

days

kHz

Base line (April 2003)*

*data published on the ANTARES site http://antares.in2p3.fr

High fluctuations in counting rates

IRFU

The Photomultiplier signal processing

A DC

A DC

Compar at or T VC

Pipe

line

T imeb ase

Dat

a fo

rmat

ting

PS D Sel

ecto

r

R ead out R e que s tT r igge r s igna l S er ial O ut put

T ime S t amp

AR

S1 P

ro

ce

ss

ing

I nt e gr at or

1 Single Photon Electron (SPE) :Charge/Time stamp = 6 bytes1 full waveform (Anode):Anode + clock samples = 263 bytes

Waveform used for detector calibration / PM signal analysis (very useful)On line trigger (Neutrino track finder) only uses SPE events.Detector operates in SPE mode~ 10 Mb/s per PM in average*900 PM = ~9 Gb/s for the full detector

ARS ASIC : Analog Ring Sampler

IRFU

Offshore: on-board system

Thermal dissipation (titanium container)

Limited room (~15 boards of 12 cm )

Limited electrical power (~35 W / storey)

Very limited access (1 / 3 years)

Numerous modules: MTBF problem( 400 spatial « satellites »)

IRFU

LCM

Detector readout : an Ethernet network

Slow control

LCM

LCM

MLCM

LCM

SCM

Data

Responsibilities :IN2P3 : Slow-control/data baseCEA : Sub-marines dataNIKHEF : Data wavelength multiplexingFor each sector (DWDM)NIKHEF : Run control/on shore acquisition

Ethernet 100 Mb/s(1 fibre WDM/45 m max)

Ethernet 1 Gb/s(2 fibres DWDM/330 m max)

JunctionBox

5x Ethernet 1 Gb/s1x Ethernet 100 Mb/s

(2 fibres DWDM/100 m)

60x Ethernet 1 Gb/s12x Ethernet 100 Mb/s

(24 fibres DWDM/40 Km)5 Sectors

12 Lines

Processor board

Switch board

Shore station

LCM ; Local Control Module

MLCM ; Master LCM

SCM : String control Module

A separate proprietary Network to time stamp the PM signals (1 ns precision)

IRFU

RTOS

SDRAM Memory(64 MB)

FlashMemory (4 MB)

Boot/ Local

FileSystem

Offshore Acquisition nodeA dedicated processor board

Processor (Motorola MPC860P@80MHz)

SlowControl

SlowControl

Task

Slow control for the storeyEthernet Link 100Mb/s

To Shore station

DataTask

Data

Data from the storey(ASIC ARS)Programmable

Logic

ProgrammableLogic

FPGA

IRFU

LCM data/slow control processing

ARS data

Logic Device (FPGA)

Mu

ltip

lexor

Buffers

Dynamic Memory1/Status2/RAZ Time3/Counters4/SPE5/Anode Waveform6/Anode Waveform+Dynode Counters

SPE

Anode Waveform

Counters

ARS 0

ARS 1

ARS 0

ARS 5

ARS 1

ARS 2

ARS 3

ARS 4

Processor

100 Mb/s Ethernet Port

104 ms Data

To shore PC 1

To shore PC 2

104 msPeriodic Slow control/on demand/configuration

104 ms Data

ClockExperiment

Slow Controlobject

IRFU

Offshore Processor board

Power 4W full charge

TCP/IP Network throughput– Running Linux/MontaVista : 25 Mb/s– Running vxWorks/windRiver : 30 Mb/s,

50 Mb/s With « Zero Copy Buffer » option

Operational Configuration– Selected RTOS vxWorks– No DAQ performances drop due to slow-

control– Measured data rate : 50 Mb/s

IRFU

Global DAQ Architecture

Off-shore :Off-shore :• 300 Detection nodes300 Detection nodes• No trigger / All data to shoreNo trigger / All data to shore• Nodes synchronised by a Nodes synchronised by a global clock (Physics event time global clock (Physics event time stamp)stamp)• Each node send 104 ms of Each node send 104 ms of data to the same On-shore data to the same On-shore processing nodeprocessing node

On-shore :On-shore :• 80 Processing nodes80 Processing nodes• Each node treat a full 104 ms Each node treat a full 104 ms detector view. Neutrino Track detector view. Neutrino Track finding.finding.• Full software triggerFull software trigger

Ethernet switch routing

Off-shore : robustness/hardware frozenOff-shore : robustness/hardware frozen

On-shore : On-shore : Processing power/upgradableProcessing power/upgradable

IRFU

On-line Trigger principle

No dead time“Self triggered” SPE signals send to shoreAll data send to shore. No hardware trigger (local storey level1 trigger implemented but not used)High fluctuations (bioluminescence) absorbed in the front end processor board Max storey rates: 120 Mb/sHigh Rate Veto applied to a PM, typically 400 kHz

IRFU

Network topology

Used as data concentratormultiples input ports one single output port

PC PC PC PC PC PC PC

ON-SHORE SWITCH: (in: 60x1000, out: 100x1000)

SECTOR SWITCH (in: 5x100, out: 1x1000) SECTOR SWITCH (in: 5x100, out: 1x1000)

LCM LCM LCM LCM LCM LCM LCM LCM LCM LCM

congestion risk intelligent control

~21 Mb/s

IRFU

Acquisition Data flow

Offshore processor

PMHigh rate Veto@400 kHz

x3

x5

Off shore Ethernet Switch x60

Ethernet Backbone x1

50 M bit/s max

Data Filter x80

Offshore

Onshore250 M bit/s max

10 Hz Neutrino Candidates (2,6 G Bytes/day)

Offline :~3 ascending Neutrino/day

Counting rate (12 G Bytes/day)

IRFU

Conclusion

No hardware trigger. All Data to shore concept (pushed by NIKHEF/DWDM)Offshore system fully configurable from shore (firmware, software, RTOS image)Onshore trigger fully upgradableConcept is working fine. 12 lines currently operating by 2500 m depth.Development time not reduced because software development and debugging may be as time consuming as hardware.