Neutron Data Acquisition

Steven Hartman BES Detector Workshop, August 2012

2 Managed by UT-Battelle for the U.S. Department of Energy

Outline

•  Introduction • Survey of neutron data acquisition systems •  Identified Needs • R&D efforts underway at SNS • Conclusion

3 Managed by UT-Battelle for the U.S. Department of Energy

Introduction: System level approach

• Data Acquisition and Instrument Control –  Interface to Detector Systems

•  Hardware and software –  Interface to beam line systems (choppers, optics, etc.) –  Interface to sample environment equipment (temperature,

pressure, magnetic field, etc.) –  Experiment automation, scripting, scanning –  User Interface

• Data Management •  Interface to Data Analysis and Visualization

4 Managed by UT-Battelle for the U.S. Department of Energy

Introduction: Functional Diagram

Acquire    Data  

Timing  

Digi0zer  

 SE,  motors,  choppers…  

Reduce  Data  

Experiment  Automa0on  

User  

Data  Storage  

Store  and  Catalog  Data    

Slow  Controls  

Analyze  and  Visualize  Data  

Publish  and  Curate  Data  

5 Managed by UT-Battelle for the U.S. Department of Energy

Introduction: Challenges

• Higher power pulsed sources, bigger detector arrays, time of flight information, event data –  Greater data rates –  Larger data sets –  More complex analysis needs

• Neutron user community –  Growing user base –  Newcomers to neutron scattering

• Need to maximize the value of the data produced

6 Managed by UT-Battelle for the U.S. Department of Energy

Survey: DAQ for Neutron Facilities

7 Managed by UT-Battelle for the U.S. Department of Energy

Survey: HFIR

• High Flux Isotope Reactor, ORNL, USA –  Reactor-based source

• SpICE (Spectrometer and Instrument Control Environment –  LabView –  Windows –  UI, instrument control,

scanning/automation, analysis capability

8 Managed by UT-Battelle for the U.S. Department of Energy

Survey: ILL

•  Institut Laue-Langevin, France –  Reactor-based source

• NOMAD –  Java UI –  C++ service –  CORBA –  Linux

9 Managed by UT-Battelle for the U.S. Department of Energy

Survey: SINQ

• Swiss Spallation Neutron Source, PSI, Switzerland –  Accelerator-based source (spallation)

• SICS and Gumtree SE –  Java UI (Eclipse RCP) –  SICS backend –  Linux

10 Managed by UT-Battelle for the U.S. Department of Energy

Survey: ISIS

•  ISIS, Rutherford Appleton Laboratory, UK –  Accelerator-based source (spallation)

•  LabView and OpenGenie –  Windows

• Software framework under review with prototype EPICS test stand under development

11 Managed by UT-Battelle for the U.S. Department of Energy

Survey: SNS

• Spallation Neutron Source, ORNL, USA –  Accelerator-based source (spallation)

• Custom software suite (currently) –  C++, Labview, Python, Windows –  Event mode data acquisition –  High data rates, will increase

• Upgrades and planning in process –  Operability and maintainability –  EPICS for instrument control and UI –  Improved data handling –  Better integration with data analysis

12 Managed by UT-Battelle for the U.S. Department of Energy

Survey: ESS

• European Spallation Source, Sweden –  Accelerator-based source (spallation) –  Anticipate operations beginning in 2019

• Conducting reviews of current neutron source DAQ systems • Data Challenges

–  Up to 300 million events per second across the instrument suite –  Data archiving estimate of ~8 PByte per year

13 Managed by UT-Battelle for the U.S. Department of Energy

Survey: Summary

•  Little commonality across neutron sources to date –  Detector interface electronics

•  Often custom designed –  Data acquisition software

•  Developed in-house –  Instrument control software

•  Often developed in-house plus some commercial software –  User Interface

•  No common look and feel

14 Managed by UT-Battelle for the U.S. Department of Energy

Survey: Options

•  Initial efforts underway to find areas for collaboration across sources for instrument control and user interface –  SNS, ISIS, ESS, . . . –  EPICS as a framework

•  Already in use for accelerator control and light-source beam line control –  Collaboration in data reduction/analysis now underway

•  Mantid Project: ISIS, SNS

15 Managed by UT-Battelle for the U.S. Department of Energy

Needs

•  2011 ASCR/BES Data Workshop •  2012 SNS/ESS Accelerating Data Analysis Workshop

16 Managed by UT-Battelle for the U.S. Department of Energy

Needs

• Data Reduction and Data Analysis in “real time” with Data Acquisition –  Reduce and analyze data as it is taken –  Data files available “instantly” at end of run

17 Managed by UT-Battelle for the U.S. Department of Energy

Needs

•  Feedback from analysis to experiment control •  Integration of modeling, experiment and data analysis

18 Managed by UT-Battelle for the U.S. Department of Energy

Research and Development at SNS

Timing and Accelerator Data

Aggregator “SMS” PreProcessor

Electronics

Electronics

Streaming Translation

Sample

Environment Equipment, Choppers,

Etc.

Scan Service Slow

Controls: EPICS

Experiment UI: CSS

Live View

Streaming Reduction

Analysis UI:

Mantid

NeXus Data: PFS

“DSP”

19 Managed by UT-Battelle for the U.S. Department of Energy

R&D: ADARA

Slow Controls Sample Environment &

Scanning Services

Detector Preprocessor(s)

(1 or more)

Manages sample environment, choppers, motors, etc.

Global High-performance

file system

Electronics

Shared Resources Per Beam Line Resources

Pushes Events + Timing

Aggregates event and environmental data

Streams data to Translation Services and Data Reduction

and Analysis

Pushes environmental data Receives controls commands for

processing and response

Stream Management Service

NeXus TranslationServices

Analysis Clusters

Data Portals

Analysis Workstations

User Interface

Hosts analysis and environmentalcontrols GUI(s)

Detectors

Streaming Data Reduction & Analysis

ServicesReceives aggregate event stream and provides live

reduction & analysis services

ADARA – Accelerating Data Acquisition, Reduction and Analysis Collaboration – NDAV (NScD), Tech Int. (NCCS), IDAC/RAD (NScD)

20 Managed by UT-Battelle for the U.S. Department of Energy

R&D: Low-level Electronics

•  Timing Receiver –  Bringing accelerator timing signal closer to the detector electronics –  Prototype system on commissioning beam line

• Beam monitors, chopper phase, fast synchronous signals –  Treat like a detector –  Stream as events

• Standardize interface boards –  FMC modules

21 Managed by UT-Battelle for the U.S. Department of Energy

R&D: Low-level Electronics

•  TCP/IP as data bus –  Need to stream data over network anyway –  Technology moving in this direction

•  Replace custom hardware with commodity components –  Working on conceptual design for UDP from FPGA to computer –  How far down?

•  FPGA vendors pushing network capabilities

22 Managed by UT-Battelle for the U.S. Department of Energy

R&D: Slow Controls and UI

• EPICS Toolkit –  In use for accelerator controls at SNS and light-sources –  Used for beam line controls at a number of light sources

• Prototype using HFIR Imaging beam line –  Simulated images and motors at this point –  Uses devices drivers previously developed for synchrotron

beam lines

23 Managed by UT-Battelle for the U.S. Department of Energy

Conclusions

•  Look at the whole data flow: acquisition, instrument control, reduction, analysis, visualization, data storage, access –  Systems engineering approach –  Optimize the whole data flow and the whole work flow –  Be aware of maintenance and support effort

• Software development is not free –  Effort required for quality software often underestimated –  Look for opportunities for collaboration

• We are experimental facilities for Users to: –  Collect the most (appropriate) data during their beam time –  Understand that data as much as possible/practical

24 Managed by UT-Battelle for the U.S. Department of Energy

Questions and Comments