CDF data production models 1

Data production models for Data production models for the CDF experimentthe CDF experimentS. Hou S. Hou for the CDF data production teamfor the CDF data production team

CDF data production models 2

CDF collaborationCDF collaborationCollider Detector experiment at the Fermilab Tevatron collider

Study collisions of1 TeV protons with 1 TeV anti-protons

CDF data production models 3

Trigger, Data AcquisitionTrigger, Data Acquisition

Sub-detector signalstrigger

CDF detector data taking rate2005

Achieved2006

upgradeTevatron luminosity : 1.8x1032 cm-2s-1 3x1032 cm-2s-1

Level-1 acceptance : 27 kHz 40 kHzLevel-2 acceptance : 850 Hz 1 kHzEvent Builder (EVB) : 850X0.2 MB/s 500 MB/sLevel-3 acceptance : 110 Hz 150 Hz

to Tape storage rate : 20 MB/s 40 MB/s

8 data logging streams Event size : ~140 kByte Average data rate ~ 5 M events/day

CDF data production models 4

Data logging rateData logging rate

Data logging rate up to Sep 2005

1 fb-1 of data recorded

Data logging rate increase w. luminosity of proton, anti-proton beamsTotal data volume increase w. integrated luminosity

Good-run raw dataFeb 2002 - Dec 2004 1017 M events = 201 k files = 185 TByteDec 2004 - Sep 2005 756 M events = 102 k files = 95 TByte

CDF data production models 5

Data flowData flow

CDF DAQProduction farm

Enstore

raw raw datasetsdatasetsraw raw datasetsdatasets

CDFAnalysisFarm

remote CAFs

User desk top

dCache

ProductionProductiondatasetsdatasetsProductionProductiondatasetsdatasets

CDF data production models 6

Data flow, Enstore storageData flow, Enstore storage

Level-3 farm

Level-1,2 Trigger, DAQ

sub-detector

Dat

aBas

e C

alib

ratio

n8 raw-datasets

52 production datasets

Run

sp

litte

r

File

ca

talo

g

Data logging is in divided by Trigger table 6 physics, 2 monitoring streams

Split events by Trigger table 52 production datasets

Enstore tape library storage 18 STK 9940B drives 200 GB/tape 30 MByte/s read/write Steady R/W rate ~1TByte/drive/day

CDF data production models 7

Computing facilityComputing facility

dCachefile-servers

10Gbit 2Gbit

Remote sites

Analysis farm

Production farm

Enstoretape library File-servers Servers

starlight

CDF Online DAQ

2Gbit

Oracle DB

offline users

CDF data production models 8

Data processing tasksData processing tasks

Raw data event reconstruction• apply detector calibration • calculate detected physics contents • output to assigned trigger datasets

One input file one binary job split output files

Concatenation of output files Raw data file is 1 GByte, Output file size varies 5 MByte to 1 GByteConcatenate small files of the same datasets

in data taking sequence to 1 GByte files

CDF data production models 9

Production farm, 1Production farm, 1stst model model

dfarm

network MySQL,DB

run-

split

ter

calib

ratio

n

Reg

iste

rco

ncat

enat

ed

55

11

22

33

worker

stager

concatenator

44

66

Reg

iste

rou

tput

Reg

iste

rin

put

Direct I/O to Enstore tape library• Custom I/O node to Enstore

FBS batch system• dfarm collection of all worker IDE buffer of input and output files

Farm Processing system• MySQL for bookkeeping• Concatenation with rigid run sequence output truncated to 1 GB files

Performance•Peak rate at 1 TB input/day used to process data up to Dec 2004

CDF data production models 10

Upgrade to CAF & SAM Data HandlingUpgrade to CAF & SAM Data Handling

Condor batch system dressed for CDF CAF (CDF Analysis Farm) package

interface for job submission and monitoring uniform platform to other CDF computing facilities compatible to distributed computing development

Data handling system SAM (Sequential Access via Metadata)

database application for file metadataprovide file locationsload files from tapes to caches

dCache (joint project of DESY+FNAL)virtualizes disk usage, loading files from tapesfiles appear to user as always on disk

CDF data production models 11

Production farm, upgrade Production farm, upgrade

output

merged4 4

fileserver

network SAM,DB

inpu

t-U

RL

run-

split

ter

calib

ratio

n

decl

are

met

adat

a

worker

22

11

33

55

dCache

Upgrade to distributed computing infrastructure:SAM data handing & Condor CAF

A CAF submit, parallel operations of - SAM Project

- Activating data handling to deliver files of the assigned SAM dataset- Tracking file consumption status

- Condor batch JOB- Consuming files of the associated SAM project- declare SAM metadata for bookkeeping

Concatenation of outputMerge output files sorted in run sequence

Store to Enstore via SAM Declare metadata and parentagefor bookeeping

CDF data production models 12

Production challengeProduction challenge

Operation tasks : “cron jobs” Resource monitoring Submission and monitoring

SAM projects Binary jobs on CAF farms

Concatenation and store

Service interface and monitoring Enstore tape I/O SAM Data handling, DB service CDF online, calibration DB, software

Timely process every event collectedInterface to Data-handing, DataBase, multiple CAFSPrecision bookkeeping on millions of files zero tolerance to error, every event is counted

CDF data production models 13

Resource MonitoringResource Monitoring

CDF DB, SAM DB, Data-HandlingCAF condor batch systemFileserver storage

Prohibited cron jobs missing required services

CDF data production models 14

CAF condor monitoringCAF condor monitoring

Tarball (archived execution binary file) distributed to worker CPUsInput files copied via SAM from dCacheEnd of job, output files are copied to assigned fileserver

CPU engagement is monitored

CDF data production models 15

Farm monitoringFarm monitoring

Worker CPUs (Ganglia)& input (rcp) waiting

Traffic to fileserver (xfs)

Bandwidth limit :Input: Enstore loading to dCacheOutput: multiple workers to fileservers 1Gbit network port to IDE: 40 MB/s1output dataset to Enstore: 30 MB/s

CDF data production models 16

SAM project monitorSAM project monitor

Input is delivered by SAM Data-Handling system Input files are organized in data-sets Each data-set is submitted to a SAM project Each project is associated with a CAF condor job

SAM projects monitoredSAM projects monitored

CDF data production models 17

Monitoring a SAM projectMonitoring a SAM project

Consumption of a data-set is monitored File delivery by SAM from registered locations (dcache, samcache, Enstore etc) Consumption by CAF worker is monitored

CDF data production models 18

Bookkeeping via SAM metadataBookkeeping via SAM metadataEach output file has a bookkeeping metadataTagging on parent-daughter after completion

Automatic recovery : on datasets having incomplete daughters

CDF data production models 19

Production stabilityProduction stabilityCAF

condor is very reliable worker hardware failure occasional RAID down-graded occasional

Service 24x7 Oracle, Enstore service SAM, dCache shift support

CPU usage total 6, output to 6 Fileserver Rougher CPU usage at the end

as streams were finishing up

CAF+Farm max=540 jobs

Farm CPU

Traffic to/from Production farm

GREEN In bits/sec BLUE Out bits/sec DARK Peak In bits/sec PINK Peak Out bits/sec

CDF data production models 20

Production rateProduction ratePeak performance:

Jobs distributed to two CAFs (Analysis & Production farm) use 540 CPU to match with 6 I/O streams 8 dCache input file servers, 6 output fileservers

uniform processing speed at 25 M events/day

3 TB input, 4 TB output /day

Integrated Output event logging

Daily file consumption

CDF data production models 21

Scaling capacityScaling capacity

At peak performance of 3 TB input, 4 TB output /day farm switch (2Gbit capacity) sees entire traffic

average load is 800 Mbit/s saturated by simultaneous network to one fileserver Gbit link (40 MB/s)

(corresponds to 100 jobs per data stream for CDF)

Scaling on CPU Add more CPU to a CAF Distribute jobs to multiple CAFs

Scaling on network I/O Limited by the 6 data-stream algorithm, split further Scale by fileservers (more Gbit links) Scale by tape drives

CDF data production models 22

Summary Summary

CDF production farm upgrade has reached a reliable rate of 3 TByte/day Capacity is scalable by increasing CPU and I/O ports

Easy and reliable operation tolerant to error recovery, with zero data loss