Upload
bodlosh
View
259
Download
2
Embed Size (px)
DESCRIPTION
38th Software Development, Ostrava 2012
Citation preview
Software development for the COMPASS
experiment
Martin Bodlák1 Vladimír Jarý1∗ Igor Konorov2
Alexander Mann2 Josef Nový1 Stephan Paul2
Miroslav Virius1
1Faculty of Nuclear Sciences and Physical EngineeringCzech Technical University in Prague
2Physik-DepartmentTechnische Universität München
Conference “Tvorba softwaru 2012”24th May 2012, Ostrava
Vladimí Jarý et al. Software development for the COMPASS experiment
Overview
1 IntroductionCOMPASS experiment
2 Current DAQ systemArchitecture of the systemDATE package
3 Control and monitoring software for a new DAQ systemMotivation and requirementsOverview of the hardware architectureLayers of the DAQ softwareImplementation detailsPerformance tests
4 Conclusion and outlook
Vladimí Jarý et al. Software development for the COMPASS experiment
COMPASS experiment
COMPASS: Common muon and proton apparatus forstructure and spectroscopyexperiment with a fixed target situated on the Super ProtonSynchrotron particle accelerator at CERN, [1]scientific program approved in 1997 by CERN
experiments with hadron beam (glueballs, Primakoffscattering, charmed hadrons,. . . )experiments with muon beam (gluon contribution to thenucleon spin, transverse spin structure of nucleons,. . . )multiple types of polarized target
data taking started in 2002plans at least until 2016 as COMPASS-II
3 programs: GPDs, Drell-Yan, Primakoff scattering
international project: cca 250 physicists from 11 countriesand 29 institutions
Vladimí Jarý et al. Software development for the COMPASS experiment
COMPASS spectrometer
polarized target on the left, length approximately 50 m
COMPASS spectrometer, image taken from [1]
spectrometer consists of detectors:1 measurement of deposited energy (calorimeters)2 particle identification (RICH, muon filters)3 particle tracking (wire chambers)
Vladimí Jarý et al. Software development for the COMPASS experiment
Terminology
event: collection of data describing flight and interactionsof particle through the spectrometerroles of the data acquisition system (DAQ):
1 reads data produced by detectors (readout)2 assembles full events from fragment (event building)3 sends events into a permanent storage (data logging)4 enables configuration, control, and monitoring (run control)5 preprocesses and filters data (e.g. track reconstruction,
online filter)
trigger : selects physically interesting events (or refusesnoninteresting events) in a high rate environment withminimal latencytrigger efficiency : ε = Ngood(selected)/Ngood(produced) < 1DAQ deadtime: D = timesystem is busy/timetotal
when system is busy, it cannot accept any other triggerwhich leads to loss of data
Vladimí Jarý et al. Software development for the COMPASS experiment
Overview of the TDAQ system
Structure of the trigger and data acquisition system according to [4]
Vladimí Jarý et al. Software development for the COMPASS experiment
Current DAQ architecture
influenced by the cycle of the SPS particle accelerator:12 s of accelaration, 4.8 s of extraction (spill/burst)key aspects: multiple layers, parelelism, buffering
1 detector (frontend) electronics:preamplify, digitize data250000 data channels
2 concentrator modules (CATCH, GeSiCA):perform readout (triggered by the Trigger Control System)append subevent header
3 readout buffers: buffering subevents in spillbuffer PCI cardsmakes use of the SPS cycle to reduce data rate to 1/3 of theonspill rate, roughly stable data rate on the output(derandomization)
4 event builders:assemble full events from subeventssend full events to the permanent storage, storemetainformation about events into the Oracle DBadditional tasks (online filter, data quality monitoring)
Vladimí Jarý et al. Software development for the COMPASS experiment
Current DAQ software
based on the ALICE DATE package[2]DATE distinguishes two kinds of processors:
1 local data concentrators (LDCs)perform readout of subevents, correspond to readout buffers
2 global data collectors (GDCs)perform event building, correspond to event builders
requirements on the nodes:1 all nodes must be x86 compatible2 all nodes must be powered by GNU/Linux OS3 all nodes must be connected to the network supporting the
TCP/IP stackflexible system (fixed targer mode × collider mode)scalable system (full scale LHC experiment × smalllaboratory system with one processor)performance:
40 GB/s readout2.5 GB/s event building1.25 GB/s storage
Vladimí Jarý et al. Software development for the COMPASS experiment
Functionality
DATE provides:1 readout, data flow2 event building3 run control4 interactive configuration (based on the MySQL database)5 event monitoring (COOOL)6 data quality monitoring (MurphyTV )7 information reporting (infoLogger, infoBrowser )8 online filter (Cinderella)9 load balancing (EDM, optional)
10 log book11 . . .
Vladimí Jarý et al. Software development for the COMPASS experiment
Problems with existing DAQ system
Motivation260 TB recorded during the 2002 Run, 508 TB during the2004 Run, more than 2 PB during the 2010 Runincreasing number of detectors and detector channels,trigger rate⇒ increasing data ratesaging of the hardware⇒ increasing failure rate of hardwarePCI technology deprecated
Main idea of the new systemreplace ROBs and EVBs by custom FPGA-based HWhardware based data flow control and event buildingsmaller number of components, higher reliability
Vladimí Jarý et al. Software development for the COMPASS experiment
Overview of the hardware architecture
frontend electronics and concentrator modules unchangedreadout buffers and event builders replaced with customhardware:
Field Programmable Gate Array (FPGA) technologyFPGA card designed as a module for AdvancedTelecommunications Computing Architecture (ATCA) carriercard, in total 8 carrier cards:
6 for data multiplexing2 for event buildingeach carrier card equipped with 4 FPGA modulesdifferent functionality, same firmware
FPGA card equipped with 4 GB of RAM, 16 serial links(bandwidth 3.25 GB/s)softcore processor on cards for powering control andmonitoring software, communication based on Ethernet
ROBs and EVBs will be used for computing farm
Vladimí Jarý et al. Software development for the COMPASS experiment
Hardware architecture
Vladimí Jarý et al. Software development for the COMPASS experiment
Requirements analysis
Requirements:distributed system, communication based on TCP/IPcompatibility with Detector Control Systemcompatibility with software for physical analysisremote control and monitoringmultiple user rolesreal time not required
Decisions:use the DIM library for communicationdo not use the DATE packagepossibly reuse some DATE components (COOOL,MurphyTV )keep data format unchanged
Vladimí Jarý et al. Software development for the COMPASS experiment
Software architecture
Roles participating in the control and monitoring software
Vladimí Jarý et al. Software development for the COMPASS experiment
Roles participating in the software
1 Master processcontrols slave processesreceives commands from GUIauthenticate and authorize usersreads and writes configuration to online database
2 Slave processesmonitor and control the hardwarereceive configuration information and commands from themaster process
3 GUIreceives information about health of the system from themaster processsends commands to the master process that distributesthese commands to the slave processes
4 Message logger: collects messages produced by otherprocesses and stores them into database
5 Message browser: displays the messages produced byother process
Vladimí Jarý et al. Software development for the COMPASS experiment
Implementation details
communication between nodes based on the DIM libraryimplementation in Qt framework
slave processes implemented in C++ language, without Qt
scripting in Python (e.g. starting of the slave processes)MySQL database (compatibility with Detector ControlSystem and DATE)complex system⇒ describe behavior of the master andthe slave processes by state machines
Vladimí Jarý et al. Software development for the COMPASS experiment
State machines
State machine describing behavior of the master process
Vladimí Jarý et al. Software development for the COMPASS experiment
DIM Library[3]
developed for the DELPHI experiment at CERNasynchronous one-to-many communication inheterogenous network environment [3]based on the TCP/IPinterfaces to C, C++, Python, Java languagescommunication between servers (publishers) and clients(subscribers) through DIM Name Server (DNS)types of messages:
services updated at regular intervalsservices updated on demandcommands
Vladimí Jarý et al. Software development for the COMPASS experiment
DIM Name Server
Position of the DIM Name Server
Vladimí Jarý et al. Software development for the COMPASS experiment
Evaluation of the system
Test scenario:number of nodes: 2 - 16message size: 100 B - 500 kBCOMPASS internal network during winter shutdown (GigabitEthernet)standard x86 compatible hardware (event builders)
Tests performed:performance
is system able to update information about status ofhardware every 100 ms?
stability
Vladimí Jarý et al. Software development for the COMPASS experiment
Results of the performance tests
Transfer speed as a function of size of the message
Vladimí Jarý et al. Software development for the COMPASS experiment
Results of the stability tests
Stability of the software in time
Vladimí Jarý et al. Software development for the COMPASS experiment
Summary and outlook
1 Analysis of the existing data acquisition systembased on the DATE packagescalability issues, deprecated technologies (PCI bus)
2 Development of control and monitoring software for newDAQ architecture
analysis of requirements on softwaredescription of the hardware architecturedefinition of roles and behavior of the systemimplementationperformance tests
3 Goals:to test system on the real hardwareto have fully functional system in 2013to deploy the system in 2014
Vladimí Jarý et al. Software development for the COMPASS experiment
The bibliography
P. Abbon et al.: The COMPASS experiment at CERN, In:Nucl. Instrum. Methods Phys. Res., A 577, 3 (2007) pp.455–518. See also the COMPASS homepage athttp://wwwcompass.cern.ch
T. Anticic et al. (ALICE DAQ Project): ALICE DAQ and ECSUser’s Guide, CERN EDMS 616039, January 2006
C. Gaspar: Distributed Information Management System[online]. 2011. Available at: http://dim.web.cern.ch
W. Vandeli: Introduction to Data Acquisition, In: InternationSchool of Trigger and Data Acquisition, Roma, February2011
Acknowledgement
This work has been supported by the MŠMT grants LA08015and SGS 11/167.
Vladimí Jarý et al. Software development for the COMPASS experiment