EGEE is a project funded by the European Union under contract IST-2003-508833

Middleware Planning for LCG/EGEE

Bob Jones

EGEE Technical Director

e-Science Grids and Web Services meeting 23th April 2004

www.eu-egee.org

e-Science Grids and Web Services meeting, 23th April 2004 - 2

Contents

• Very Brief overview of EGEE & LCG Objectives and status How they relate to each other The ARDA project

• Middleware planning Organisation Key middleware deliverables and milestones Strategy being adopted Current Status

• Summary

e-Science Grids and Web Services meeting, 23th April 2004 - 3

EGEE manifesto:Enabling Grids for E-science in Europe

 Applications

Geant network

Grid infrastructure

• Goal Create a wide European Grid production

quality infrastructure on top of present and future EU RN infrastructure

• Build On: EU and EU member states major investments in Grid Technology International connections (US and AP) Several pioneering prototype results Large Grid development teams in EU require

major EU funding effort

• Approach Leverage current and planned national and

regional Grid programmes Work closely with relevant industrial Grid

developers, NRENs and US-AP projects

e-Science Grids and Web Services meeting, 23th April 2004 - 4

EGEE Partners

• Leverage national resources in a more effective way for broader European benefit

• 70 leading institutions in 28 countries, federated in regional Grids

e-Science Grids and Web Services meeting, 23th April 2004 - 5

EGEE Applications

• EGEE Scope : ALL-Inclusive for academic applications (open to industrial and socio-economic world as well)

• The major success criterion of EGEE: how many satisfied users from how many different domains ?

• 5000 users (3000 after year 2) from at least 5 disciplines

• Two pilot applications selected to guide the implementation and certify the performance and functionality of the evolving infrastructure: Physics & Bioinformatics

e-Science Grids and Web Services meeting, 23th April 2004 - 6

EGEE Project Structure

JRA1: Middleware Engineering and Integration

JRA2: Quality Assurance

JRA3: Security

JRA4: Network Services Development

SA1: Grid Operations, Support and Management

SA2: Network Resource Provision

NA1: Management

NA2: Dissemination and Outreach

NA3: User Training and Education

NA4: Application Identification and Support

NA5: Policy and International Cooperation

24% Joint Research 28% Networking

48% ServicesEmphasis in EGEE is on operating a productiongrid and supporting the end-users

32 Million Euros EU funding over 2 years starting 1st April 2004

e-Science Grids and Web Services meeting, 23th April 2004 - 7

EGEE and LCG

EGEE builds on the work of LCG to establish a grid operations service• LCG: a worldwide collaboration of

The LHC experiments The Regional Computing Centres Physics institutes

• Mission: Prepare and deploy the computing environment that will be used by the

experiments to analyse the LHC data• Strategy:

Integrate thousands of computers at dozens of participating institutes worldwide into a global computing resource

Rely on software being developed in advanced grid technology projects, both in Europe and in the USA

• Status: LCG service up and running with LCG-2 mware – successfully being

used for LHC data challenges

e-Science Grids and Web Services meeting, 23th April 2004 - 8

• Service opened on 15 September 2003

– with 12 sites

• Middleware package components from

European DataGrid (EDG)

US Virtual Data Toolkit (Globus,

Condor, PPDG, iVDGL, GriPhyN)

• About 30 sites by the end of the year

Upgraded version of the grid software

(LCG-2) in February 2004

• VO for D0 managed by NIKHEF

• Hewlett Packard to provide

“Tier 2-like” services for LCG,

initially in Puerto Rico

LCGThe LCG Service

06-Apr-04country centre country centre

Canada TRIUMF, Vancouver

Spain PIC, Barcelona

Czech Republic

Prague IFIC, Valencia

France IN2P3, Lyon IFCA, SantanderGermany FZK, Karlsruhe University of

BarcelonaDESY Uni. Santiago de

CompostelaHolland NIKHEF,

AmsterdamCIEMAT, Madrid

Hungary Budapest UAM, MadridItaly CNAF, Bologna Switzerland CERN

INFN, Torino CSCS, MannoINFN, Milano Taiwan Academia Sinica,

TaipeiINFN, Roma NCU, TaipeiINFN, Legnaro UK RAL

Japan ICEPP, Tokyo Cavendish, Cambridge

Poland Krakow Imperial, London

Russia SINP, Moscow USA FNALBNL

Regional Centres Connected to the LCG Grid

e-Science Grids and Web Services meeting, 23th April 2004 - 9

ARDA ProjectCollaborationCoordinationIntegration

SpecificationsPrioritiesPlanning

EGEE Middleware Activity

ALICEDistr.

analysis

ATLASDistr.

analysis

CMSDistr.

analysis

LHCbDistr.

analysis

Distributed Physics AnalysisThe ARDA Project

ARDA – distributed physics analysis batch to interactive end-user emphasis

• 4 pilots by the LHC experiments (core of the HEP activity in EGEE NA4)

• Rapid prototyping pilot service

• Providing focus for the first products of the EGEE middleware

• Kept realistic by what the EGEE middleware can deliver

e-Science Grids and Web Services meeting, 23th April 2004 - 10

EGEE Implementation

• From day 1 (1st April 2004)Production grid service based on the LCG infrastructure running LCG-2 grid

middleware (SA)

LCG-2 will be maintained until the new generation has proven itself (fallback solution)

• In parallel develop a “next generation” grid facility (JRA)Produce a new set of grid services according to evolving standards (Web Services)

Run a development service providing early access for evaluation purposes

Will replace LCG-2 on production facility in 2005

Globus 2 based Web services based

EGEE-2EGEE-1LCG-2LCG-1

EDGVDT . . .

LCG

EGEE

. . .AliEn

e-Science Grids and Web Services meeting, 23th April 2004 - 11

JRA1: Middleware Engineering and Integration

• ObjectivesProvide robust, supportable middleware componentsIntegrate grid services to provide a consistent functional basis for the

EGEE grid infrastructureVerify the middleware forms a dependable and scalable infrastructure

that meets the needs of a large, diverse eScience user community 5 partners, approx 16% of total project budget

• 4 main development clusters: UK CERN IT/CZ Nordic (security, JRA3)

e-Science Grids and Web Services meeting, 23th April 2004 - 12

Milestones and Deliverables for 2004

MonthDeliverables

&Milestones

ItemLead Partner

Status

Jun’04 MJRA1.1 Tools for middleware engineering and integration deployed

CERN Most tools identified (SCM)Issue: CNRS person tbi

Jun’04 DJRA1.1 (Document) Architecture and Planning (Release 1)

CERN Design document as starting point; writing organized for May

Jun’04 MJRA1.2 Software cluster development and testing infrastructure available

CERN Done for CERN/UK/IT/CZNordic to be finalized

Aug’04 MJRA1.3 Integration and testing infrastructure in place including test plans (Release 1)

CERN Sites identified; teams established;

Aug’04 DJRA1.2 (Document) Design of grid services (Release 1)

CERN Design document as starting point

Dec’04 MJRA1.4 Software for the Release Candidate 1 CERN Work on prototype started; will evolve into 1st release candidate

M08 – Amsterdam conference: Tech preview of release candidate 1 available

e-Science Grids and Web Services meeting, 23th April 2004 - 13

Towards a Prototype

• Design team formed in December 2003• Started service design based on component breakdown defined by the

LCG ARDA RTAG• Leverage experiences and existing components from AliEn, VDT, and

EDG.• A working document - Overall design & API’s• Basis for architecture (DJRA1.1 – June ‘04) and design (DJRA1.2 –

Aug’04) document• Aim to have first prototype ready by end of April

Focus on key services; exploit existing components Initially an ad-hoc installation at CERN and Wisconsin Open only to a small user community Expect frequent changes (also API changes) based on user feedback and

integration of further services• Enter a rapid feedback cycle

Continue with the design of remaining services Enrich/harden existing services based on early user/operations feedback

e-Science Grids and Web Services meeting, 23th April 2004 - 14

Prototype Evolution

• Evolution of the prototype Envisaged status at end of 2004:

• Key services need to fulfill all requirements (application, operation, quality, security, …) and form a deployable release

• Remaining services available as prototype Will develop a roadmap

• Incremental changes to prototype (where possible)• Early user feedback through ARDA and early deployment on SA1 pre-

production service• Detailed release plan being developed

e-Science Grids and Web Services meeting, 23th April 2004 - 15

Guiding Principles

• Lightweight existing services Easily and quickly deployable

• Interoperability Allow for multiple implementations

• Resilience and Fault Tolerance

• Co-existence with deployed infrastructure Run as an application

• Service oriented approach Follow and contribute production expertise to standardization efforts No mature WSRF implementations exist to date, hence: start with plain WS

– WSRF compliance is not an immediate goal Review situation end 2004

e-Science Grids and Web Services meeting, 23th April 2004 - 16

Summary

• LCG is currently providing a production grid service using the LCG-2 software

• EGEE has started 1st April The first project conference was held in Cork this week and showed

that all activities are up and running

• Middleware EGEE will develop a new set of web services based grid

middleware• An early prototype will be tested in May• Due to timing issues, the first release will probably only be based on

basic web services• EGEE will be involved in the grid standardisation process and is keen to

adopt such standards as they become stable