David Abbott - Jefferson Lab DAQ group

Data Acquisition Development at JLABData Acquisition Development at JLAB

DAQ Group now stands at 5 members. Recent Experiments have begun to test limits of the

current distribution of CODA (v 2.5). Aging technologies (software and hardware) are being

retired and replaced both for support of the 6 GeV program as well as development of 12 GeV.

Continue to use open standards and minimize the use of commercial software while maximizing use of commercial hardware.

We continue to focus on a “migration” plan from CODA2 to CODA3 (v 2.6).

Data Acquisition Status

General DAQ Issues…

Front-end hardware is evolving. Real-time intelligence is moving from the CPU to FPGAs. Old hardware technologies are no longer commercially supported (FASTBUS).

CPU-Based real-time readout on a per event basis limits the maximum accepted L1 trigger rate (~10 KHz).

32 crate limit on the trigger distribution system is nearly reached in Hall B. Event transport limitations in the current CODA architecture are being seen

for moderately complex systems. Computing platform and OS changes (Muli-core, more memory, 64 bit

systems etc…) are not taken advantage of. Aging software technologies and reliance on third party packages are

making code portability and upkeep difficult. Monitoring and control of large numbers of distributed objects are not

handled in a consistent way (too many protocols). “Slow” controls only minimally supported

CODA3 - Requirements/Goals Pipelined Electronics (FADC, TDC)

– Dead-timeless system– Replacement for obsolete electronics– Eliminate large numbers of delay cables

Integrated L1/L2 Trigger and Trigger Distribution System– Support up to 200 KHz L1 Trigger– Use FADC for L1 trigger input– Support 100+ crates

Parallel/Staged Event Building– Handle ~100 of input data streams– Scalable (>1 GByte/s) aggregate data throughput

L3 Online Farm– Online (up to x10) reduction in data to permanent storage

Integrated Experiment Control – DAQ RunControl + “Slow” control/monitoring– Distributed, scalable, and “intelligent”

Proposed GlueX DAQ System

Current DAQ Projects

Components:Components: CODA Objects CODA ROC CODA EMU (EB/ER/ANA) Run Control

Software Tools:Software Tools: cMsg ET EVIO Config and Display GUIs

Hardware:Hardware: FADC/F1TDC Trigger Interface (VME/PCI) Trigger/Clock DistributionCommercial Module Support

R&D:R&D: Embedded Linux Experiment Control Staged/Parallel Event Building 200KHz Trigger/readout Clock distribution L3 Farm

Front-End Systems

VME CPUVME CPU -(MV6100) PPC, GigE, vxWorks(GE V7865) Intel, GigE, Linux CODA ROC Readout ~160-200 MB/s

Trigger InterfaceTrigger Interface - (V3)

Pipeline TriggerEvent BlockingClock distributionEvent ID Bank Info

F1 TDCF1 TDCFlash ADCFlash ADC

R&D to support fully pipelined crates capable of 200 KHz trigger rates

VXS - L1 Trigger

VME CPU -???Intel, GigELinux CODA ROCVME Readout of Event Data

Switch Sum and Trigger Switch Sum and Trigger Distribution Modules (VXS)Distribution Modules (VXS)

Collect Sums/HitsPass Data to Master L1Clock distributionTrigger Distribution

Flash ADCFlash ADC

Use VXS High speed serial backplane (P0) to collect Energy sum and hit data from FADCs

Flash ADCFlash ADC

P0P0

Building a DAQ System

ROC

ROC

ROC

ROC

To EMUTo EMUTo FileTo FileTo ETTo ET

To UserTo User

Process

Event Distribution

ET provides efficient transport of Data for building, and provides flexible User access

EMU provides easy configuration, and User specific processing options

Staged/Parallel Event Building

• Divide total throughput into N streams (1GB/sec -> N*xMB/sec).Divide total throughput into N streams (1GB/sec -> N*xMB/sec).• Two stages - Data Concentration -> Event Two stages - Data Concentration -> Event BuildingBuilding..• Each EMU is a software component running on a separate host.Each EMU is a software component running on a separate host.

AFECS - Integrated Experiment ControlAFECS - Integrated Experiment Control

FIPA Java-Based (v 1.5) “Intelligent” agents Extensions provide runtime “distributed” Containers (JVM). Agents provide a customizable intelligence (state machine)

and communication (cMsg, CA, SNMP etc…) with external processes.

Many independent “logical” control systems can operate within the platform.

System is scalable. Agents can migrate to JVM containers on different nodes at runtime. System tested: 3 Containers on different hosts with 1000

Agents controling 1000 physical components distributed over 20 other nodes.

~40% CPU and 200 MB usage for each JVM

CODA ROC

CODA EMU

EPICS IOC 1

EPICS CAG

Trigger soft

Trigger hard

Online ANA

A

A

A

A

A

A

A

S

S

S

S

S

NR NA NCPhysical Components

Normative Agents

Supervisor agent

Grand supervisor

Front-End

ACC

AFECS Platform(Java 1.5+)

IPC

IPC

WEB

IPC

Hierarchy of Control

IPC

CODA EvolvesCODA is a software toolkit from which data acquisition systems with varying degrees of complexity can be built.

A typical system might look

Single Board Computer

FASTBUSVMECAMAC

ROC rcServer

EB

ROC

Network: Ethernet FDDI ATM

UNIX/LINUX

DISK/TAPEET

User Proc.

RunControlGUI

ER

cdev

dp_tcl

mSQL Database

msqld cmlogServer

cmlogDatabase

cMsg

AFECS

CODA3

CODA 2.6 Features

• Integrate new AFECS Run Control System• cMsg IPC replaces RC<->(ROC,EB,ER) communication as

well as CMLOG message logging.• Various Support for newer operating systems and compilers

– vxWorks 5.5, 6+– RHEL 4, Solaris 10, OS X

• New and Updated tools – ET upgraded, 64bit compliant– Db2cool– EVIO package

• Support for new CODA3 Objects and components• Integration of long time bug fixes, new driver libraries and

feature enhancements

Summary

The DAQ Group must support ALL experimental programs at JLAB. The current group must grow by at least 2 FTEs soon to manage current timelines.

CODA 2.6 is available now, and will provide an integration path for CODA 3 technologies.

Much DAQ software development is dependent on custom hardware development in order satisfy many 12GeV requirements.

Current DAQ projects reflect the philosophy that we can progress to support the physics of the 12 GeV program through an evolution of the existing proven system.