DataGrid Applications Federico Carminati WP6 WorkShop December 11, 2000

  • View
    213

  • Download
    0

Embed Size (px)

Text of DataGrid Applications Federico Carminati WP6 WorkShop December 11, 2000

  • DataGrid ApplicationsFederico CarminatiWP6 WorkShopDecember 11, 2000

    DataGrid WP6 Workshop*11 December 2000

    The distributed computing modelAssumptionsRaw data will be kept at CERN (backup if affordable)Tier 1 will have ~10% of raw dataReconstruction will be done at CERN (2 passes)ESD (aka DST) and TAG will be shipped to Tier1-2Simulation will all be done in the Tier1-2, data will be shipped to CERN (in which form? Raw, ESD, AOD)The Tier1-2 will process ESD to produce AOD (formerly aka n-tuple) as many time as necessary, producing their own TAGUsers will access the ESD/AOD/TAG remotely

    DataGrid WP6 Workshop*11 December 2000

    The distributed computing modelBasic principleEvery physicist should have in principle equal access to the data and to the resourcesThe system will be extremely complexNumber of components in each siteNumber of sitesDifferent tasks performed in parallel: simulation, reconstruction, scheduled and unscheduled analysis

    DataGrid WP6 Workshop*11 December 2000

    WP 8 philosophyDefine a common upper middle-layer of GRID services common to the 4 experimentsCommon API for common tasksFile replicaJob submission and monitoring.Collaborate to define a common set of requirements and milestones, also with WP 9/10Share the same testbeds and facilities for data challengesIntroduce a user view in the project

    DataGrid WP6 Workshop*11 December 2000

    WP8 plans and requirementsTestbed Release 0 (1Q 2001)A working, standard, installation of Globus, at CERN and at other labs.Standard recipes for file transfer and job submission (at CERN, this installation will probably be interfaced to LSF).A contact point for GLOBUS and later DataGRID software in each lab incase of problemsA clear policy for "experiment-wide" authorisation to allow testing across national boundaries.This will only work if enough support is provided to usersWP8 activities in general require to have a substantial part of the GRID services available since as soon as possible

    DataGrid WP6 Workshop*11 December 2000

    WP8 plans and requirementsTestbed release 1 (3Q 2001)Distributed user autentication and resource allocation or pre-allocation, as dynamic allocation can come later.Distributed data dictionary (location of files on different servers).Basic network configuration, monitoring and diagnostic tools.Basic monitoring and diagnostic of a cluster of PC's.Distributed scheduling for the jobs that are submitted in a coordinated way (this does not include the "chaotic job activity" coming from isolated users which should be addressed in the subsequent release).Access to the basic information about job status and errors.Guidelines for configurating farms of PC's with fast disk access.

    DataGrid WP6 Workshop*11 December 2000

    WP8 plans and requirementsTestbed release 2 (3Q 2002)Replica management and network optimised trasfer of data from different file systems.Tools for configurating farms of PC's with fast disk access, monitoring their status and for automathized s/w installation and management.A prototype of scheduling and load balancing for chaotic analysis jobsBasic functionality for dynamic resource allocationBasic functionality for job partitioning

    DataGrid WP6 Workshop*11 December 2000

    WP8 plans and requirementsTestbed release 3 (3Q 2003)Scheduling and balancing of chaotic analysisTools to ensure "robustness" and error recovery of the system

    DataGrid WP6 Workshop*11 December 2000

    WP8 Wkshop Nov 16ConclusionsWP8 non-experiment-specific personpowerNeeded to implement WP8 common policiesCERN person identified and hired (I.Augustin)Should come from the funded contribution of each partner (CNRS, CERN, PPARC, NIKHEF and INFN) as a share of the 60 funded person-monthsA message sent to the PMB for each partner to identify this personpower

    DataGrid WP6 Workshop*11 December 2000

    WP8 Wkshop Nov 16ConclusionsTwo kinds of Test-bed participation.Formal participation defined by WP6Informal commitment to install Datagrid provided tools, and participate in tests. Kick-start and Update kitsExperiments to provide WP8 with installation and upgrade kitsWP8 personpower will install them in the WP6 test-beds locationsKits have to coexist on the same machines without interferenceLocations participating but not in WP6 will provide their own personpower for the installationThis activity will be coordinated by the CERN WP8 person (I.Augustin)

    DataGrid WP6 Workshop*11 December 2000

    WP8 Wkshop Nov 16ConclusionsCollection of requirementsPresent WP8 requirements judged vague by other WP's (rightly!)Other WP's should have asked WP8/9/10 questions they didntMeeting of December 1st of the ATF was rather inconclusiveNew strategy: WP8-10 to produce a three tiered documentShort term use casesLong term use casesGeneral requirementsWP8-10 also to produce pilot applicationsDecember 15 ATF should consolidate user requirementsDataGRID Workshop on January 15US experts invited to discuss user requirements and first proposition of architecture

    DataGrid WP6 Workshop*11 December 2000

    WP8 Wkshop Nov 16ConclusionsSet up a technical WP8 technical WGone application software expert from each experiment and from ESA (WP9) and biology (WP10)the WP8 experiment-neutral personpower in the different partners (5 people)Chaired by the CERN WP8 person acting as WP8 architect)

    DataGrid WP6 Workshop*11 December 2000

    WP8 Wkshop Nov 16ConclusionsMain tasks Help WP8 architect to collect the requirements from the experiment and ESA and Biology for the ATF Liaise with the Middleware WP's (1-5) regarding the services required by the applications, and the definition of the appropriate interfaces.Discuss WP8 architectural questionsDo WP8-10 require a common 'upper middleware' layer over middleware services?Do they want to interface to generic middleware services directly? Definition of the 'sample' applicationsCMS and LHCb have existing running distributed applicationsALICE is following and ATLAS will see whether it can add itsDefine requirements for the Test-bed and Networking

    DataGrid WP6 Workshop*11 December 2000

    WP8 Wkshop Nov 16ConclusionsMain tasks (cont)Provide technical liaison with the Test-bed and Networking workpackages (e.g. attend their meetings)Special priority points are, for instance Provision of Standard, Supported, Documented Globus installation kit Definition of Test-bed sites and contact people Definition of Certification system to be employed by the project Compact sub-groupWill meet, virtually or in person, in its own right, preparing our point of view for the global Test-bed and Architecture meetingsExperiment TB reps, unless the same person, would normally not attend, unless required for a forthcoming TB meetingThe frequency of meetings decided by WP8 architect and coordinator.

    DataGrid WP6 Workshop*11 December 2000

    ConclusionsActivity of WP8 well startedMain concerns areArchitecture design seems to start with difficultyWe do not have an architect yetCoordination with WP9-10 not yet very effectiveNeed better communication with WP1-5Need to start interacting more effectively with WP6-7