5 March 2002 DG PARIS Jesus Marco CSIC IFCA(Santander) Development of GRID environment for interactive applications J.Marco (CSIC) DATAGRID WP6 MEETING

5 March 2002 DG PARIS

Jesus Marco CSIC IFCA(Santander)

Development of GRID environment for

interactive applications

J.Marco (CSIC) DATAGRID WP6 MEETING (PARIS) 5-III-2002



The CROSSGRID project

• EU Vth Framework Programme (IST)– 3 year project, starting date 1st March 2002– 4.86 M € contract signed last week!

• Objectives:– Final user interactive applications– Extend GRID into 11 EU countries

in close collaboration with DATAGRID

• 21 Partners:– COORD: CYFRONET (PL, coordinator M.Turala)– CR: UvA (N), PSNC(PL), FZK (D) ,CSIC (E) – Companies: Algosystems(GR) + DATAMAT (I) – ICM,INP,IPJ (PL) + IISAS (SK) + USTUT+TUM (D) + ULINZ (A) +

TCD(Ir) UAB+USC (E) + LIP (P) + Demo+AuTH (Gr) + UCY (Cy)



CROSSGRID Collaboration

Poland:Cyfronet & INP CracowPSNC PoznanICM & IPJ Warsaw

Portugal:LIP Lisbon

Spain:CSIC SantanderValencia & RedIrisUAB BarcelonaUSC Santiago & CESGA

Ireland:TCD Dublin

Italy:DATAMAT

Netherlands:UvA Amsterdam

Germany:FZK KarlsruheTUM MunichUSTU Stuttgart

Slovakia:II SAS Bratislava

Greece:AlgosystemsDemo AthensAuTh Thessaloniki

Cyprus:UCY Nikosia

Austria:U.Linz



Project Structure

• Workpackages:– Applications (P.Sloot, UvA)– Programming Environment (H.Marten,

FZK)– New Services & tools (N.Meyer, PSNC)– Testbed (J.Marco, CSIC)– Management (M.Turala)

• Architecture Team – Lead by M.Bubak, Cyfronet



Applications

• Final user interactive applications:– Response loop in the range of seconds to minutes

• Computing & Data intensive• Architecture basis: GLOBUS + DATAGRID• Multidisciplinary:

– Simulation & visualisation for surgical procedures– Air pollution combined with weather forecasting– Distributed Data Analysis in HEP – Flooding crisis team decision support system



Tools & Services

• GRID Application Programming Environment– Verification of MPI use– Performance prediction– Monitoring

• Services & Tools– User friendly portals– Roaming access– Efficient distributed data access– Specific resource management

Open Architecture approach



Testbed

• Testing and validation for– Applications– Programming environment– New services & tools

• Emphasis on collaboration with DATAGRID + extension to DATATAG

• Extension of GRID across Europe



Extension of GRID



Application development

• Common requirements:–Near real-time performance across the network–Efficient access to large distributed databases–Large scale parallel simulations

–Data discovery–Data mining–Visualisation

• Deployment on the CROSSGRID testbed will test the applications in the final user environment and provide feedback

WP2WP3MPI

Adressed in collaboration between WP1 tasks



Distributed Physics Analysis in HEP

• Final user applications for interactive physics analysis in a GRID aware environment for LHC experiments

• Acess to large O/R DBMS through middleware server vs access to catalog + distributed root data files

• Distributed data-mining techniques (mainly Neural Networks, also self organizing maps, ...)

• Integration of user-friendly interactive acess via portals

• CSIC + UAB + FZK + INP + ICM + IPJ



HEP Interactive Physics Analysis

MPIMaster

Scalability: OK in local cluster…



WP4: International Testbed Organisation

• Objective:– Provide a distributed resource facility– Support testbed sites across Europe, integrating national

facilities– Assure interoperability with DATAGRID testbed

• 15 testbed sites across 9 countries• ~16 FTE working on them• Start points:

– Join DataGRID testbed gradually– Use GEANT for network connectivity– Extend testbed with incremental releases as applications and

middleware demand and provide feedbackEffort coordinated by CSIC, will benefit from experience in

DATAGRID testbed also in TCD, UAB, LIP



Testbed MAP

Auth Thessaloniki

UvA Amsterdam

FZK Karlsruhe

TCD Dublin

UAB Barcelona

LIP Lisbon

CSIC IFIC Valencia

CSIC Madrid

USC SantiagoCSIC IFCA Santander

DEMO Athens UCY Nikosia

CIFRONET Cracow

II SAS Bratislava

PSNC Poznan

ICM & IPJ Warsaw



Network (Geant) setup

CrossGrid WP4 - International Testbed Organisation

UCY NikosiaDEMO Athens

Auth Thessaloniki

CYFRONET Cracow

ICM & IPJ Warsaw

PSNC Poznan

II SAS Bratislava

FZK Karlsruhe

UvA Amsterdam

CSIC Valencia

UAB Barcelona

CSIC Santander

CSIC Madrid

LIP Lisbon

USC Santiago

TCD Dublin



Tasks in WP4

4.0 Coordination and management

(task leader: J.Marco, CSIC, Santander)–Coordination with WP1,2,3–Collaborative tools (web+videoconf+repository)–Integration Team

4.1 Testbed setup & incremental evolution (task leader:R.Marco, CSIC, Santander)

–Define installation–Deploy testbed releases–Trace security issues


Testbed site responsibles:– CYFRONET (Krakow) A.Ozieblo– ICM(Warsaw) W.Wislicki– IPJ (Warsaw) K.Nawrocki– UvA (Amsterdam) D.van Albada– FZK (Karlsruhe) M.Kunze– IISAS (Bratislava) J.Astalos– PSNC(Poznan) P.Wolniewicz– UCY (Cyprus) M.Dikaiakos– TCD (Dublin) B.Coghlan– CSIC (Santander/Valencia) S.Gonzalez– UAB (Barcelona) G.Merino– USC (Santiago) A.Gomez– Demo (Athenas) C.Markou– AuTh (Thessaloniki) D.Sampsonidis– LIP (Lisbon) J.Martins



Tasks in WP4

4.2 Integration with DATAGRID (task leader: M.Kunze, FZK)–Coordination of testbed setup–Exchange knowledge–Participate in WP meetings

4.3 Infrastructure Support (task leader: J.Salt, CSIC, Valencia)–Fabric management–HelpDesk–Provide Installation Kit–Network support

4.4 Verification & quality control (task leader: J.Gomes, LIP)–Feedback –Improve stability of the testbed




Workflow and interfaces


Setup

report onrequirements

& planning

4.0 Coordination and management (PM1-36)

Plan Evolution: Prototype 0 FINAL

WP 1,2,3 WP 1,2,3 WP 1

requirementsfrom

1st prototypereleaset

final testbed

3 15 21 33 36

D4.5

D4.1

M4.1

30

D4.7

testbedsetupon selectedsites

6

D4.2

D4.4

Feedback from

Feedbackmainly from

PU

PUM4.3

PU

internalprogressreport

CO CO

D4.6


CO

M4.4

final demo+ report

PU

PU

First testbed Support Prototype 1

IntegrationTeam

10


DATAGRIDtestbed

Feedbackfrom

D4.3

CO

M4.2



Questions…

• Interactive use of resources:– Priorities/Allocation/Booking policies

• How to allocate 500 machines for 5 minutes of interactive use?

– “Connection”: • Concept of Interactive Session?• Use of Permanent and Volatile storage

– Scheduler? • Strongly data-location aware

– Dynamic versioning/installation?• Software configuration for different applications



Testbed Organization:

• Use national certificates as in DataGrid:– Spain, Portugal, Ireland, Germany, Poland– Greece, Cyprus, Slovakia…

• V.O.:– Define CROSSGRID testbed V.O.– Join DataGrid HEP V.O. (Alice,Atlas,CMS,LHCb)– Define new V.O. for other applications:

• Med V.O.• Flood Prevention V.O.• Meteo V.O.



Testbed Organization:

• Kick-off meeting in Krakow (17th March)

• First three months:– Review and setup local infraestructure– Gather requirements from

applications/middleware– Setup HelpDesk, define Integration Team– Reinforce connection with Datagrid:

• Extend testbed (Germany,Poland,Portugal,Spain)• Find topics for contribution

Documents

5 March 2002 DG PARIS Jesus Marco CSIC IFCA(Santander) Development of GRID environment for interactive applications J.Marco (CSIC) DATAGRID WP6 MEETING