DataGrid Applications Federico Carminati WP6 WorkShop December 11, 2000

DataGrid Applications

Federico Carminati

WP6 WorkShop

December 11, 2000

DataGrid WP6 Workshop

211 December 2000

The distributed The distributed computing modelcomputing model

AssumptionsRaw data will be kept at CERN – (backup if affordable)

Tier 1 will have ~10% of raw data

Reconstruction will be done at CERN (2 passes)

ESD (aka DST) and TAG will be shipped to Tier1-2

Simulation will all be done in the Tier1-2, data will be shipped to CERN (in which form? Raw, ESD, AOD)

The Tier1-2 will process ESD to produce AOD (formerly aka n-tuple) as many time as necessary, producing their own TAG

Users will access the ESD/AOD/TAG remotely


311 December 2000

The distributed The distributed computing modelcomputing model

Basic principleEvery physicist should have in principle equal access to the data and to the resources

The system will be extremely complexNumber of components in each site

Number of sites

Different tasks performed in parallel: simulation, reconstruction, scheduled and unscheduled analysis


411 December 2000

WP 8 philosophyWP 8 philosophy

Define a common upper middle-layer of GRID services common to the 4 experiments

Common API for common tasksFile replicaJob submission and monitoring….

Collaborate to define a common set of requirements and milestones, also with WP 9/10Share the same testbeds and facilities for data challengesIntroduce a user view in the project


511 December 2000

WP8 plans and WP8 plans and requirementsrequirements

Testbed Release 0 (1Q 2001)A working, standard, installation of Globus, at CERN and at other labs.Standard recipes for file transfer and job submission (at CERN, this installation will probably be interfaced to LSF).A contact point for GLOBUS and later DataGRID software in each lab incase of problemsA clear policy for "experiment-wide" authorisation to allow testing across national boundaries.

This will only work if enough support is provided to usersWP8 activities in general require to have a substantial part of the GRID services available since as soon as possible


611 December 2000


Testbed release 1 (3Q 2001)Distributed user autentication and resource allocation or pre-allocation, as dynamic allocation can come later.

Distributed data dictionary (location of files on different servers).

Basic network configuration, monitoring and diagnostic tools.

Basic monitoring and diagnostic of a cluster of PC's.

Distributed scheduling for the jobs that are submitted in a coordinated way (this does not include the "chaotic job activity" coming from isolated users which should be addressed in the subsequent release).

Access to the basic information about job status and errors.

Guidelines for configurating farms of PC's with fast disk access.


711 December 2000


Testbed release 2 (3Q 2002)Replica management and network optimised trasfer of data from different file systems.

Tools for configurating farms of PC's with fast disk access, monitoring their status and for automathized s/w installation and management.

A prototype of scheduling and load balancing for chaotic analysis jobs

Basic functionality for dynamic resource allocation

Basic functionality for job partitioning


811 December 2000


Testbed release 3 (3Q 2003)Scheduling and balancing of chaotic analysis

Tools to ensure "robustness" and error recovery of the system


911 December 2000

WP8 Wkshop Nov 16WP8 Wkshop Nov 16ConclusionsConclusions

WP8 non-experiment-specific personpowerNeeded to implement WP8 common policies

CERN person identified and hired (I.Augustin)

Should come from the funded contribution of each partner (CNRS, CERN, PPARC, NIKHEF and INFN) as a share of the 60 funded person-months

A message sent to the PMB for each partner to identify this personpower


10

11 December 2000


Two kinds of Test-bed participation.Formal participation defined by WP6

Informal commitment to install Datagrid provided tools, and participate in tests.

Kick-start and Update kitsExperiments to provide WP8 with installation and upgrade kits

WP8 personpower will install them in the WP6 test-beds locations

Kits have to coexist on the same machines without interference

Locations participating but not in WP6 will provide their own personpower for the installation

This activity will be coordinated by the CERN WP8 person (I.Augustin)


11

11 December 2000


Collection of requirementsPresent WP8 requirements judged vague by other WP's (rightly!)Other WP's should have asked WP8/9/10 questions – they didn’tMeeting of December 1st of the ATF was rather inconclusive

New strategy: WP8-10 to produce a three tiered documentShort term use casesLong term use casesGeneral requirements

WP8-10 also to produce pilot applicationsDecember 15 ATF should consolidate user requirementsDataGRID Workshop on January 15

US experts invited to discuss user requirements and first proposition of architecture


12

11 December 2000


Set up a technical WP8 technical WGone application software expert from each experiment and from ESA (WP9) and biology (WP10)

the WP8 experiment-neutral personpower in the different partners (5 people)

Chaired by the CERN WP8 person acting as WP8 architect)


13

11 December 2000


Main tasks Help WP8 architect to collect the requirements from the experiment and ESA and Biology for the ATF Liaise with the Middleware WP's (1-5) regarding the services required by the applications, and the definition of the appropriate interfaces.Discuss WP8 architectural questions

Do WP8-10 require a common 'upper middleware' layer over middleware services?Do they want to interface to generic middleware services directly?

Definition of the 'sample' applicationsCMS and LHCb have existing running distributed applicationsALICE is following and ATLAS will see whether it can add its

Define requirements for the Test-bed and Networking


14

11 December 2000


Main tasks (cont)Provide technical liaison with the Test-bed and Networking workpackages (e.g. attend their meetings)Special priority points are, for instance

1. Provision of Standard, Supported, Documented Globus installation kit 2. Definition of Test-bed sites and contact people 3. Definition of Certification system to be employed by the project

Compact sub-groupWill meet, virtually or in person, in its own right, preparing our point of view for the global Test-bed and Architecture meetingsExperiment TB reps, unless the same person, would normally not attend, unless required for a forthcoming TB meetingThe frequency of meetings decided by WP8 architect and coordinator.


15

11 December 2000

ConclusionsConclusions

Activity of WP8 well startedMain concerns are

Architecture design seems to start with difficulty

We do not have an architect yet

Coordination with WP9-10 not yet very effective

Need better communication with WP1-5

Need to start interacting more effectively with WP6-7

Documents

DataGrid Applications Federico Carminati WP6 WorkShop December 11, 2000