View
218
Download
3
Embed Size (px)
Citation preview
DAQ for Test&Commissioningat Point 2
Klaus Schossmaier
CERN PH-AID
ALICE TPC Collaboration Meeting
Heidelberg, Germany
12-13 February 2004
TPC Collaboration meeting, 12-13 Feb 2004
2DAQ Test&Commissioning at Point 2
Planning Parameters
System: DAQ system for Test&Commissioning
(= LHCC milestone)
Size: 12 DDLs for TPC8 DDLs for others
Place:LHC Point 2
Date: Q1 2005
People: ALICE DAQ team
TPC Collaboration meeting, 12-13 Feb 2004
3DAQ Test&Commissioning at Point 2
DAQ Installation at Point 2
SXL HallSX Hall
UX25 (Experimental Area)
PX24/CR1 (DAQ)
PX24/CR2 (HLT)
SR Hall
Access Shaft
DDLLAN
(ALICE sub-detector assembly)(Networking)
PX24/CR3 (DCS)PX24/CR4 (Misc.)
LAN
ACR WR1 WR2
TPC Collaboration meeting, 12-13 Feb 2004
4DAQ Test&Commissioning at Point 2
Installation at PX24/CR1
10 LDCs (Local Data Concentrator):
rackmount PCsU1, U2, U4 heightequipped with 20 D-RORC cards DDL readout, sub-event building
HLT interface
4 GDCs (Gobal Data Collector):
rackmount PCsU1 height equipped with FC cards full-event building
1 DSS (DAQ Server Services):
rackmount PCU4 heighthot-swap SCSI disks RunControl, Logger
On-line Monitoring
1 KVM switch
1 FC switch
1 TDS (Transient Data Storage):
rackmount disk arrayU2 heightIDE of FC disks event buffering
GE switches
PDS (Permanent Data Storage)
CERN computer center
20 DDLs
DDL patch
panels
HLT (HighLevel Trigger)
PX24/CR2
TPC Collaboration meeting, 12-13 Feb 2004
5DAQ Test&Commissioning at Point 2
Installation Issues
Hardware/Mechanical issues: Equipment: rackmount PCs, switches, disk array, … Rack assembly, DDL installation, cabling, … Infrastructure: power supply, networking, cooling, …
Software/Testing issues: Operating System: Linux + customizations DAQ Software: DATE + configuration Standalone tests Integration tests
Reference System in our AID lab: small, complete DAQ system with 6 DDLs to spot problems in advance to tackle problems during operation
TPC Collaboration meeting, 12-13 Feb 2004
6DAQ Test&Commissioning at Point 2
Reference Setup – Front
KVM switch:
Raritan Paragon UMT2161
4x LDCs:
- 1U, 2U, 4U height- dual Xeon- 6x D-RORC cards DDL
2x GDCs:
- 1U height- dual Xeon- Qlogic QLA2310F cards
2x disk arrays:
- Infortrend IFT-6330
- DotHill SANnet II- Fibre Channel
DSS:
- 4U height- quad Xeon- 3x 36GB SCSI disks
L3 rack
2x GB Ethernet switch:
3COM SuperStack 3
Fibre Channel switch:
Broacade SilkWorm 3800
DDLs:
- to the DDL lab
TPC Collaboration meeting, 12-13 Feb 2004
7DAQ Test&Commissioning at Point 2
Reference Setup - Rear
Power distributor
Cat5 cables:
- Ethernet- KVM- RJ45 connectors
Fiber Channel cable:
- 2 Gbit/s multimode - LC-LC connectors
L3 rack
Mounting rails (!)
TPC Collaboration meeting, 12-13 Feb 2004
8DAQ Test&Commissioning at Point 2
DDL & D-RORC
DDL link card
Uniform half-CMC card for both SIU (= Source Interface Unit) and DIU (= Destination Interface Unit)
2.125 Gbit/s components (Agilent) for 200 MB/s bandwidth
bridge up to 200 meters
DDL patch panel with D-RORC
Two D-RORC types: 1 DDL link card plugged in, 2 integrated DDL channels
PCI 64 bit, 66 MHz
ALTERA FPGA, IP core from PLDA
200 MB/s throughput with DATE
TPC Collaboration meeting, 12-13 Feb 2004
9DAQ Test&Commissioning at Point 2
Rackmount PCs
Evaluation/Benchmarking item LDC GDC DSS
Rackmount height 1U, 2U, 4U 1U 4U
CPU chipsetmotherboard
dual Xeon 2.4 GHzE7501
Supermicro X5DPE-G2
dual Xeon 2.66 GHzE7501
Supermicro X5DPI-G2
quad Xeon 1.5 GHzServerWorks GC-HE
Intel SRSH4
Main memory 2 GBDDR 266 MHz
2 GBDDR 266 MHz
4 GBDDR 266 MHz
Ethernet dual Gigabit Ethernet on-board
dual Gigabit Ethernet on-board
Gigabit and Fast Ethernet on-board
Disk 40 GB IDE 40 GB IDE hot-swap 36 GB SCSI disks
PCI slots up to 6 PCI-X for D-RORC cards
1 PCI-Xfor Fibre Channel card
6 PCI-X2 PCI 5V
Linux installation Red Hat 7.3 Red Hat 7.3 Red Hat 7.3 (rather old)
TPC Collaboration meeting, 12-13 Feb 2004
10DAQ Test&Commissioning at Point 2
Disk Array Systems
front view
(IFT-6330)
rear view
(IFT-6330)
Infortrend IFT-6330-12F2D:
12x 300 GB IDE-based disks
dual 2 Gbit/s Fibre Channel ports
Embedded manager via front panel, RS232C, or IP
2x logical volumes: each one has1.1 TB, RAID5, with 5 disks
DotHill SANnet II 200:
12x 36.7 GB FC-based disks
dual 2 Gbit/s Fibre Channel ports
RAID configurations via RS232C
TPC Collaboration meeting, 12-13 Feb 2004
11DAQ Test&Commissioning at Point 2
DAQ Software: DATE
DATE (= Data Acquisition and Test Environment)is the software framework of the ALICE DAQ.
Runs on Linux platforms (e.g. Red Hat 7.3) Released (latest is version 4.7) and documented DATE evolves with requirements and technology
Features/Packages: Dataflow: DDL readout, event building System configuration and Run control Monitoring: on-line performance, on/off-line data quality Utilities: message logger, control/configure front-end electronics, …
TPC Collaboration meeting, 12-13 Feb 2004
12DAQ Test&Commissioning at Point 2
Readout with DATE
TPC RCU 3 with SIU
D-RORC with DIU
DDL
FEC
LDC
Step 0: Hardware is in place
Step 1: Run Linux with the physmem and the rorc driver
Step 2: Install DATE with a general configuration for the LDC
Step 3: Set up configuration file to enter parameters like D-RORC serial number, channel number, page size, checks, etc.
Step 4: Start/Stop the acquisition
Step 5: Analyse the raw data files, e.g. /tmp/run123.000, /tmp/run123.001, …
TPC Collaboration meeting, 12-13 Feb 2004
13DAQ Test&Commissioning at Point 2
DATE Run Control
TPC Collaboration meeting, 12-13 Feb 2004
14DAQ Test&Commissioning at Point 2
FEE Configuration/Control
TPC RCU 3 with SIU
D-RORC with DIU
DDL
FEC
LDC
Step 0: Hardware is in place
Step 1: Run Linux with the the rorc driver on the LDC
Step 2: Write a script (ASCII file) for the FeC2 interpreter
Step 3: Run this script, e.g. type FeC2 123 example.fec2
# send command, 19 bit information
write_command 0x10f
# read status, 19 bit address, 19 bit reply
read_and_print 0x100 “Register XYZ: 0x%x”
# download data block, 19 bit address, file
write_block 0x600 pedestal.hex “%x”
# verify data block, 19 bit address, file
read_and_check_block 0x600 pedestal.hex “%x”
stop_if_failed -1
S/N of D-RORC
TPC Collaboration meeting, 12-13 Feb 2004
15DAQ Test&Commissioning at Point 2
On-line Monitoring
MOOD Framework: DATE + ROOT environment Interfaces to detector code
MOOD Applications: Visualization, e.g. TPC sector
test data (Hall 167) Data integrity Detector performance
Execution: DSS machine
TPC Collaboration meeting, 12-13 Feb 2004
16DAQ Test&Commissioning at Point 2
DAQ Performance
ALICE Computing Data Challenge:
1750 MB/s event building without recording, 5 days
280 MB/s event building and recording to CASTOR, 7 days
D-RORC readout of one DDL channel with DATE:
206 MB/s readout only
118 MB/s readout with event building
TPC Collaboration meeting, 12-13 Feb 2004
17DAQ Test&Commissioning at Point 2
DAQ Installations
DAQ System Location DDL LDC GDC Networking Storage
Development / Evaluation AID lab 2 4 4 Gb Ethernet disk
DDL Development DDL lab 2 1 0 Gb Ethernet disk
Detector Groups (e.g. SDD, HMPID, TOF, Muon, TRD)
Institutes 1 1 0 Fast/Gb Ethernet Disk
TPC Sector Test § PS T10 2 1 0 Fast Ethernet disk or CASTOR
ALICE Data Challenges CERN IT 0 80 80 Gb Ethernet TDS + CASTOR
Reference Setup AID lab 6 4 2 Gb Ethernet TDS
Test&Commissioning Point 2 20 10 4 Gb Ethernet TDS + CASTOR
Final System Point 2 395 185 50 Gb Ethernet TDS + CASTOR
§ Given that the RCU 3 is fully functional in time, see Joachim’s presentation.
TPC Collaboration meeting, 12-13 Feb 2004
18DAQ Test&Commissioning at Point 2
Summary + Future Work
Objective: DAQ system for Test&Commissioning at Point 2 Surface tests of ALICE sub-detectors (e.g. TPC sector) Operational in Q1 2005
Installation Layout: 20 DDL links between the PX24/CR1 counting room and the
SXL mounting hall L3 racks: DDL patch panels, 15 rackmount PCs, 1 disk array,
switches for Gigabit Ethernet, Fiber Channel and KVM DAQ Reference System in the AID lab
Future work concerning TPC and DAQ: Bergen: Development of RCU 3 firmware for the DDL interface Luciano’s lab: Integration test of RCU 3 with the DDL TPC sector test at T10§: DAQ chain with RCU 3 + DDL + DATE