November 11-17 SC06 Tampa F.Fanzago

CRAB

a user-friendly tool for

CMS distributed analysis

Federica Fanzago INFN-PADOVA

for CRAB team

November 11-17 SC06 Tampa F.Fanzago 2

• CMS (Compact Muon Solenoid) is one of the four particle physics experiment that will collect data at LHC (Large Hadron Collider) starting in 2007 at CERN.

• CMS will produce a big quantity of data

• Data should be stored and made

available for analysis to world-wide distributed physicists.

CMS overview

• ~2 PB events/year (startup luminosity 2x1033 cm-2 s-1)

• All events will be stored into files– O(10^6) files/year

• Files will be grouped in Fileblocks – O(10^3) Fileblocks/year

• Fileblocks will be grouped in Datasets

– O(10^3) Datasets (total after 10 years of CMS)

– 0.1- 100 TB

November 11-17 SC06 Tampa F.Fanzago 3

Why the grid…

ISSUES: • How to manage and where to store this huge quantity of data?• How to assure data access to physicists of CMS collaboration?• How to have enough computing power for processing and data

analysis?• How to ensure resources and data availability?• How to define local and global policy about data access and resources?

SOLUTION:• CMS will use a distributed architecture based on grid infrastructure to

ensure remote resources availability and to assure remote data access to authorized user (belonging to CMS Virtual Organization).

• The grid infrastructure guarantees also enough computing power for simulation, processing and analysis data.

November 11-17 SC06 Tampa F.Fanzago 4

CMS computing model

Online system

Tier 0

Tier 1

Tier 2

Tier 3

Offline farm

. .

Tier2 Center Tier2 Center Tier2 Center

InstituteB InstituteA

. . .workstation

Italy Regional Center

Fermilab Regional Center

France Regional Center

recorded data

The CMS offline computing system is arranged in four Tiers which are geographically distributed

Remote data accessible

via grid

CERN Computer center

November 11-17 SC06 Tampa F.Fanzago 5

• During data acquisition data from detector which got over different trigger level will be sent, stored and first step reconstructed at Tier-0.

• Then they will be distributed over some Tiers depending on the kind of physics data

• Until real data are not available, the CMS community needs simulated data to study the detector response, the foreseen physics interaction and to get experience with management and analysis of data. So a large number of simulated data are produced and distributed among computing centers.

Data distribution…

November 11-17 SC06 Tampa F.Fanzago 6

grid middleware…

Resource Broker (RB)

Workload Management

System

SE

SE

SE

UIJob submission

tools

UIJob submission

toolsData location

systemData location

system

InformationService

collector

InformationService

collector

Query for dataQuery for matchmaking

CECE

CE

Main LCG middleware components:

Virtual Organizations (CMS ...)Resource Broker (RB)Replica Catalog (LFC)

Computing Element (CE)Storage Element (SE)

Worker node (WN)User Interface (UI)

LCG middleware:

Tools for accessing distributed data and resources are provided by the World LHC Computing Grid (WLCG) that takes care about different grid flavours as LCG/gLite in Europe and OSG in the US.

November 11-17 SC06 Tampa F.Fanzago 7

Analysis in a local environment…

But now data and resources are distributed

• User writes his own analysis code and configuration parameter card– Starting from CMS specific analysis software

– Builds executable and libraries

• He apply the code to a given amount of events, whose location is known, splitting the load over many jobs– But generally he is allowed to access only local data

• He writes wrapper scripts and uses a local batch system to exploit all the computing power– Comfortable until data you’re looking for are sitting just by your side

• Then he submits all by hand and checks the status and overall progress

• Finally collects all output files and store them somewhere

November 11-17 SC06 Tampa F.Fanzago 8

…and in a distributed environment

The distributed analysis is a more complex computing task because it assume to know:

• which data are available• where data are stored and how to access them• which resources are available and are able to comply with analysis

requirements• grid and CMS infrastructure details

Users don't want deal with these kind of problem

They want to analyze data in “a simple way” as in local environment

November 11-17 SC06 Tampa F.Fanzago 9

Distributed analysis chain

To allow distributed analysis the CMS collaboration is developing some tools interfaced with available grid services, that include:

• Installation of CMS software via grid on remote resources• Data transfer service: to move and manage a large flow of data

among tiers• Data validation system: to ensure data consistency• Data location system: catalogues to keep track of data available in

each site and to allow data discovery– Dataset Bookkeeping System. It knows which data exist and

contains CMS specific description of event data– Data Location Service. It knows where data are stored. Mapping

between file-blocks and SE– Local file catalog: physical location of local data on remote SE

• CRAB: Cms Remote Analysis Builder...

November 11-17 SC06 Tampa F.Fanzago 10

CRAB CMS Remote Analysis Builder • CRAB is a user-friendly tool whose aim is to simplify the work of users

with no knowledge of grid infrastructure to create, submit and manage job analysis into grid environments. – written in python and installed on UI (grid user access point)

• Users have to develop their analysis code in a interactive environment decide which data to analyze and how to manage jobs output

• Data discovery on remote resources, resources availability, status monitoring and output retrieval of submitted jobs are fully handled by CRAB

UI withCRAB

User analysis code as in local environment

data in remote and distributed sites

November 11-17 SC06 Tampa F.Fanzago 11

CRAB workflow

datasetn.of eventsuser code

DataLocationSystem

SE

LocalFile

Catalog

DataBookkeeping

SystemJob

jdl tgz sh

WMS

Job

CE

WNWN ...

Job output

Job output

CRAB

UI

SEs list

data

Main CRAB functionalities:Input data discovery: the list of sites (SEs

name) where data are stored, querying “data location system” (DBS-DLS)

Packaging of user code: creation of a tgz archive with user code and parameters

Job creation:Wrapper of user code executable to run on WN (sh script)Jdl file: SE name as requirement to drive resources matchmakingJob splitting according to user request

Job submission to the gridJob status monitoring and output

retrievalHandling of user output: copy to UI or to

a generic Storage ElementJob resubmission in case of failure

November 11-17 SC06 Tampa F.Fanzago 12

CRAB usage (1)

Job submitted from worldwide distributed UI

The total number of jobs submitted to the grid using CRAB is

more than 1 million by 40-50 users.

SC03PTDR

SC04

More or less 1500 jobs

are submitted each day.

November 11-17 SC06 Tampa F.Fanzago 13

CRAB usage (2)

~75% job success rate (success means jobs arrive to remote sites and produce outputs) ~25% aborts due to site setup problem or grid services failure.

physicists are using CRAB to analyze

remote data stored in LCG and OSG sites.

~7000 Datasets available for O(10^8) total events, full MC

production

November 11-17 SC06 Tampa F.Fanzago 14

• CRAB tool is used to analyze remote data and also to continuously test CMS Tiers to prove the whole infrastructure robustness

• CRAB proves a CMS user with no knowledge about grid infrastructure are able to use grid services

• CRAB demonstrates distributed analysis works in a distributed environment.

• The future code development will be related to split CRAB in a client-server system with the aim to minimize the user effort to manage analysis jobs and obtain their results.

http://cmsdoc.cern.ch/cms/ccs/wm/www/Crab/

Conclusion