5 March 2002 DG PARIS Jesus Marco CSIC IFCA(Santander) Development of GRID environment for...

5 March 2002 DG PARIS

Jesus Marco CSIC IFCA(Santander)

Development of GRID environment for

interactive applications

J.Marco (CSIC) DATAGRID WP6 MEETING (PARIS) 5-III-2002

5 March 2002 DG PARIS

Jesus Marco CSIC IFCA(Santander)

The CROSSGRID project

• EU Vth Framework Programme (IST)– 3 year project, starting date 1st March 2002– 4.86 M € contract signed last week!

• Objectives:– Final user interactive applications– Extend GRID into 11 EU countries

in close collaboration with DATAGRID

• 21 Partners:– COORD: CYFRONET (PL, coordinator M.Turala)– CR: UvA (N), PSNC(PL), FZK (D) ,CSIC (E) – Companies: Algosystems(GR) + DATAMAT (I) – ICM,INP,IPJ (PL) + IISAS (SK) + USTUT+TUM (D) + ULINZ (A) +

TCD(Ir) UAB+USC (E) + LIP (P) + Demo+AuTH (Gr) + UCY (Cy)

5 March 2002 DG PARIS

Jesus Marco CSIC IFCA(Santander)

CROSSGRID Collaboration

Poland:Cyfronet & INP CracowPSNC PoznanICM & IPJ Warsaw

Portugal:LIP Lisbon

Spain:CSIC SantanderValencia & RedIrisUAB BarcelonaUSC Santiago & CESGA

Ireland:TCD Dublin

Italy:DATAMAT

Netherlands:UvA Amsterdam

Germany:FZK KarlsruheTUM MunichUSTU Stuttgart

Slovakia:II SAS Bratislava

Greece:AlgosystemsDemo AthensAuTh Thessaloniki

Cyprus:UCY Nikosia

Austria:U.Linz

5 March 2002 DG PARIS

Jesus Marco CSIC IFCA(Santander)

Project Structure

• Workpackages:– Applications (P.Sloot, UvA)– Programming Environment (H.Marten,

FZK)– New Services & tools (N.Meyer, PSNC)– Testbed (J.Marco, CSIC)– Management (M.Turala)

• Architecture Team – Lead by M.Bubak, Cyfronet

5 March 2002 DG PARIS

Jesus Marco CSIC IFCA(Santander)

Applications

• Final user interactive applications:– Response loop in the range of seconds to minutes

• Computing & Data intensive• Architecture basis: GLOBUS + DATAGRID• Multidisciplinary:

– Simulation & visualisation for surgical procedures– Air pollution combined with weather forecasting– Distributed Data Analysis in HEP – Flooding crisis team decision support system

5 March 2002 DG PARIS

Jesus Marco CSIC IFCA(Santander)

Tools & Services

• GRID Application Programming Environment– Verification of MPI use– Performance prediction– Monitoring

• Services & Tools– User friendly portals– Roaming access– Efficient distributed data access– Specific resource management

Open Architecture approach

5 March 2002 DG PARIS

Jesus Marco CSIC IFCA(Santander)

Testbed

• Testing and validation for– Applications– Programming environment– New services & tools

• Emphasis on collaboration with DATAGRID + extension to DATATAG

• Extension of GRID across Europe

5 March 2002 DG PARIS

Jesus Marco CSIC IFCA(Santander)

Extension of GRID

5 March 2002 DG PARIS

Jesus Marco CSIC IFCA(Santander)

Application development

• Common requirements:–Near real-time performance across the network–Efficient access to large distributed databases–Large scale parallel simulations

–Data discovery–Data mining–Visualisation

• Deployment on the CROSSGRID testbed will test the applications in the final user environment and provide feedback

WP2WP3MPI

Adressed in collaboration between WP1 tasks

5 March 2002 DG PARIS

Jesus Marco CSIC IFCA(Santander)

Distributed Physics Analysis in HEP

• Final user applications for interactive physics analysis in a GRID aware environment for LHC experiments

• Acess to large O/R DBMS through middleware server vs access to catalog + distributed root data files

• Distributed data-mining techniques (mainly Neural Networks, also self organizing maps, ...)

• Integration of user-friendly interactive acess via portals

• CSIC + UAB + FZK + INP + ICM + IPJ

5 March 2002 DG PARIS

Jesus Marco CSIC IFCA(Santander)

HEP Interactive Physics Analysis

MPIMaster

Scalability: OK in local cluster…

5 March 2002 DG PARIS

Jesus Marco CSIC IFCA(Santander)

WP4: International Testbed Organisation

• Objective:– Provide a distributed resource facility– Support testbed sites across Europe, integrating national

facilities– Assure interoperability with DATAGRID testbed

• 15 testbed sites across 9 countries• ~16 FTE working on them• Start points:

– Join DataGRID testbed gradually– Use GEANT for network connectivity– Extend testbed with incremental releases as applications and

middleware demand and provide feedbackEffort coordinated by CSIC, will benefit from experience in

DATAGRID testbed also in TCD, UAB, LIP

5 March 2002 DG PARIS

Jesus Marco CSIC IFCA(Santander)

Testbed MAP

Auth Thessaloniki

UvA Amsterdam

FZK Karlsruhe

TCD Dublin

UAB Barcelona

LIP Lisbon

CSIC IFIC Valencia

CSIC Madrid

USC SantiagoCSIC IFCA Santander

DEMO Athens UCY Nikosia

CIFRONET Cracow

II SAS Bratislava

PSNC Poznan

ICM & IPJ Warsaw

5 March 2002 DG PARIS

Jesus Marco CSIC IFCA(Santander)

Network (Geant) setup

CrossGrid WP4 - International Testbed Organisation

UCY NikosiaDEMO Athens

Auth Thessaloniki

CYFRONET Cracow

ICM & IPJ Warsaw

PSNC Poznan

II SAS Bratislava

FZK Karlsruhe

UvA Amsterdam

CSIC Valencia

UAB Barcelona

CSIC Santander

CSIC Madrid

LIP Lisbon

USC Santiago

TCD Dublin

5 March 2002 DG PARIS

Jesus Marco CSIC IFCA(Santander)

Tasks in WP4

4.0 Coordination and management

(task leader: J.Marco, CSIC, Santander)–Coordination with WP1,2,3–Collaborative tools (web+videoconf+repository)–Integration Team

4.1 Testbed setup & incremental evolution (task leader:R.Marco, CSIC, Santander)

–Define installation–Deploy testbed releases–Trace security issues

CrossGrid WP4 - International Testbed Organisation

Testbed site responsibles:– CYFRONET (Krakow) A.Ozieblo– ICM(Warsaw) W.Wislicki– IPJ (Warsaw) K.Nawrocki– UvA (Amsterdam) D.van Albada– FZK (Karlsruhe) M.Kunze– IISAS (Bratislava) J.Astalos– PSNC(Poznan) P.Wolniewicz– UCY (Cyprus) M.Dikaiakos– TCD (Dublin) B.Coghlan– CSIC (Santander/Valencia) S.Gonzalez– UAB (Barcelona) G.Merino– USC (Santiago) A.Gomez– Demo (Athenas) C.Markou– AuTh (Thessaloniki) D.Sampsonidis– LIP (Lisbon) J.Martins

5 March 2002 DG PARIS

Jesus Marco CSIC IFCA(Santander)

Tasks in WP4

4.2 Integration with DATAGRID (task leader: M.Kunze, FZK)–Coordination of testbed setup–Exchange knowledge–Participate in WP meetings

4.3 Infrastructure Support (task leader: J.Salt, CSIC, Valencia)–Fabric management–HelpDesk–Provide Installation Kit–Network support

4.4 Verification & quality control (task leader: J.Gomes, LIP)–Feedback –Improve stability of the testbed

CrossGrid WP4 - International Testbed Organisation

5 March 2002 DG PARIS

Jesus Marco CSIC IFCA(Santander)

Workflow and interfaces

CrossGrid WP4 - International Testbed Organisation

Setup

report onrequirements

& planning

4.0 Coordination and management (PM1-36)

Plan Evolution: Prototype 0 FINAL

WP 1,2,3 WP 1,2,3 WP 1

requirementsfrom

1st prototypereleaset

final testbed

3 15 21 33 36

D4.5

D4.1

M4.1

30

D4.7

testbedsetupon selectedsites

6

D4.2

D4.4

Feedback from

Feedbackmainly from

PU

PUM4.3

PU

internalprogressreport

CO CO

D4.6

internalprogressreport

CO

M4.4

final demo+ report

PU

PU

First testbed Support Prototype 1

IntegrationTeam

10

internalprogressreport

DATAGRIDtestbed

Feedbackfrom

D4.3

CO

M4.2

5 March 2002 DG PARIS

Jesus Marco CSIC IFCA(Santander)

Questions…

• Interactive use of resources:– Priorities/Allocation/Booking policies

• How to allocate 500 machines for 5 minutes of interactive use?

– “Connection”: • Concept of Interactive Session?• Use of Permanent and Volatile storage

– Scheduler? • Strongly data-location aware

– Dynamic versioning/installation?• Software configuration for different applications

5 March 2002 DG PARIS

Jesus Marco CSIC IFCA(Santander)

Testbed Organization:

• Use national certificates as in DataGrid:– Spain, Portugal, Ireland, Germany, Poland– Greece, Cyprus, Slovakia…

• V.O.:– Define CROSSGRID testbed V.O.– Join DataGrid HEP V.O. (Alice,Atlas,CMS,LHCb)– Define new V.O. for other applications:

• Med V.O.• Flood Prevention V.O.• Meteo V.O.

5 March 2002 DG PARIS

Jesus Marco CSIC IFCA(Santander)

Testbed Organization:

• Kick-off meeting in Krakow (17th March)

• First three months:– Review and setup local infraestructure– Gather requirements from

applications/middleware– Setup HelpDesk, define Integration Team– Reinforce connection with Datagrid:

• Extend testbed (Germany,Poland,Portugal,Spain)• Find topics for contribution