Addressing the Software and Hardware Trade-Offs of an Embedded Distributed System The Case of the ANTARES Data Acquisition and Detector Control S. Anvar,

Addressing the Software and Addressing the Software and Hardware Trade-Offs of an Hardware Trade-Offs of an

Embedded Distributed Embedded Distributed SystemSystem

The Case of the ANTARES Data Acquisition and Detector Control

S. Anvar, CEA Saclay DapniaOn behalf of the Antares collaboration

2

RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control

http://antares.in2p3.fr

A Neutrino TelescopeA Neutrino Telescope

3


The Antares CollaborationThe Antares Collaboration

Institutes Particle physics Astronomy Sea science

4


ConstraintsConstraints

Environmental constraints Baseline data rate of ~100 kHz / PMT Data rate fluctuations ~2MHz / PMT due to bioluminescence Strong uncertainties on environment parameters Reduced space and thermal conductivity …

Reliability (MTBF) Maximize simplicity Minimize consumption Minimize number of components …

Deployment constraints High cost of submarine cables Limits on string cable rigidity or sea operations …

5


Design principleDesign principle

Stringcontrol

Junctionbox

Electro-mechanicalcable

Electro-opticalcable (50 km)

Detectors

Slow controlDataClockEnergy

Processingnodes

Acquisitionnodes

OffshoreOnshore12 strings25 detecting nodes per

string (every 12.5m)3 optical modules (PMTs)

per acquisition node2 "ARS" digitizing chips per

optical module

Real time processing: Offshore: readout 1800

ARS chips spread out over 30 000 000 m3 @ 2500m underwater

Onshore: perform real time trigger computations on incoming data.

6


Each offshore acquisition node: one DAQ/SC Each offshore acquisition node: one DAQ/SC BoardBoard

ARS readout/SCLocal trigger

106 gatesXilinx FPGA



Real-timeembedded CPU

PPC MPC860VxWorks


PPC MPC860VxWorks

SDRAMMemory

64 MB

SDRAMMemory

64 MB

DAQ/SC Board

Achieved power consumption of ~5W including optical Ethernet tranceiver laser

Ethernet 100

Serial links(RS485, RS232)

7


DAQ/SC BoardARS chip

The ARS chip and the DAQ/SC boardThe ARS chip and the DAQ/SC board

8


Network Topology (1)Network Topology (1)

Cable rigidityCable diameter

Connection complexity

Each

LC

M h

as it

s ow

n I/O

dat

a flo

w(S

low

Con

trol

& P

hysi

cs D

ata)

SCM

Sectors

M

SCM

M

To Shore(DWDM)

100 Mb/s

1 G

b/s

9



10


Offshore switch Board

Allayer chip

The offshore switch boardThe offshore switch board

11


The electronic crateThe electronic crate

12


Used as data concentratormany input ports one output port


Solution barrel shifting synchronized by global clock

PC PC PC PC PC PC PC

ON-SHORE SWITCH: (in: 60x1000, out: 20x1000)

SECTOR SWITCH (in: 5x100, out: 1x1000) SECTOR SWITCH (in: 5x100, out: 1x1000)

N N N N N N N N N N

risks of congestion intelligent flow control in node

~15 Mb/s

Instant ~6 Gb/s

13


Massively Distributed and Heterogeneous Massively Distributed and Heterogeneous SystemSystem

Final System: A Network of ~320 Nodes ~300 Offshore Nodes (MPC860 + VxWorks) ~20 Onshore Nodes (PC + Linux)

A number of distributed applications running in parallel and communicating DAQ L2 Trigger Slow Control / Acoustic positioning acquisition L1 trigger ? Run control

14


Software TechnologiesSoftware Technologies

Programming Languages C/C++ for offshore software components C++ / Java for onshore processing Java / C++ for onshore display

Distribution Multiple messaging and data servers (ControlHost) TCP/IP based VxWorks Zero-copy TCP/IP (~25Mb/s ~38 Mb/s per node) Offshore network traffic control (barrel shifting)

Data and configuration storage Oracle database Root files

Design & Development Object Paradigm UML (Unified Modeling Language) + Extensions Model-driven engineering

15


Interrupt – shared memory communicationInterrupt – shared memory communication

ARS readout(FPGA)

intrHandler(RTOS)

frameSendertask (RTOS)

Semaphore(RTOS)

Data Memory(SDRAM)

write

read andwrite

Actual frameformatting andsending throughTCP/IP socket

16


Interrupt – shared memory communicationInterrupt – shared memory communicationseen as asynchronous method callseen as asynchronous method call

ARS_Read

+fillBuffers(id:int)

frameSender

1

6

FrameSender

+acq(in frameID:int)

acq(in frameID:int)

:ARS_Readout :FrameSender

6

<<fpga>>AR_Node

readout

Interrupt - SharedMem

1

<<RTOS>>FS_Node

framer

FrameSenderARS_Readout

<<reside>><<reside>>

asynchronous method call

reference to design pattern

17


Applying the design pattern: model Applying the design pattern: model transformationtransformation

ARS_Read

+fillBuffers(id:int)

frameSender

1

6

FrameSender

+acq(in x:Frame)

Implemented in VHDLImplemented in VHDL

IntrSharedMemProxy

+doIntr(in x:PrmRefType)

FrameSenderSkeleton

+accept(in x:Frame)

IntrSharedMemSkeleton

#run()#installInterrupt()+accept(in pr:PrmRefType)

PrmRefType

{bind(Frame)}

RealTime::Thread

#run() reusablesoftware

1

18


Widespread beliefWidespread belief

Performance goes against abstraction & Performance goes against abstraction & flexibilityflexibility

Performance goes against abstraction & Performance goes against abstraction & flexibilityflexibility

Wrong if the issue is explicitly addressedWrong if the issue is explicitly addressed

ActuallyActuallyAbstraction Abstraction modularity modularity specific optimization specific optimization perf. perf.

Wrong if the issue is explicitly addressedWrong if the issue is explicitly addressed

ActuallyActuallyAbstraction Abstraction modularity modularity specific optimization specific optimization perf. perf.

19


Evolutions (1)Evolutions (1)






PPC MPC860VxWorks


PPC MPC860VxWorks

SDRAMMemory

64 MB

SDRAMMemory

64 MB

DAQ/SC Board

Ethernet 100




SystemOn Chip

Xilinx Virtex II Pro?Xilinx Virtex IV



SystemOn Chip

Xilinx Virtex II Pro?Xilinx Virtex IV

SDRAMMemory

64 MB

SDRAMMemory

64 MB

DAQ/SC Board

Ethernet 100


Lower consumptionReduced number of components

increased reliability

20



Present scheme: « all data to shore »Unexpected variable baseline rates: ~100 400 kHz instead of ~70

21



Present scheme: « all data to shore »Unexpected variable baseline rates: ~100 400 kHz instead of ~100

Software L1 trigger Reduced L1 partial data sent from offshore to central L1 workstation Central L1 computes trigger with global vision of detector Trigger decision broadcasted through software Expected gain in data rate: x1000

From 5 to 30 cm/s (!)From 5 to 30 cm/s (!)

22


ConclusionsConclusions

Slow controlDataClockEnergy

HEP DAQ and trigger techniques High data rates Very weak signal Highly distributed system Soft real time constraints …

Challenges unusual for HEP No known beam (calibration) Very embedded system Very hostile environment Uncontrolled factors (biolum) …

Both Robust and Flexible Design Allows adaptation Allows for remote evolution …

Documents

Addressing the Software and Hardware Trade-Offs of an Embedded Distributed System The Case of the ANTARES Data Acquisition and Detector Control S. Anvar,