Upload
dwayne-harris
View
215
Download
0
Tags:
Embed Size (px)
Citation preview
EGI: European Grid Infrastructure
Steven Newhouse
Interim EGI.eu Director
EGI-InSPIRE Project Director
Technical Director EGEE-III1
European e-Infrastructure
• European Data Grid (EDG)– Explore concepts in a testbed
• Enabling Grid for E-sciencE (EGEE)– Moving from prototype to production
• European Grid Infrastructure (EGI)– Routine usage of a sustainable e-infrastructure
2
What is e-Infrastructure?
• Resources linked by high speed networks– Compute, Storage, Instruments, ...
• Controlled access to shared resources– Authentication, Authorisation, Accounting, ...
• Dependable services for others to use– Driven by availability and reliability metrics
• Services that are there for the long-term– Supporting experiments lasting decades
Swiss Grid Day 3
The EGEE-III Project17,000 users
139,000 LCPUs (cores)
25Pb disk
39Pb tape
12 million jobs/month
+45% in a year
268 sites
+5% in a year
48 countries
+10% in a year
162 Virtual Organisations
+29% in a year
Over 20 active communities in 112 VOs
EGICollaboration
NGI
NGI
NGI
NGI
ResearchCommunity
ResearchCommunity
ResearchCommunity
ResearchCommunity
EIRO
EIRO
EGI.eu
ResearchCommunity
5
EGI.eu
• Coordination for European Grid resources– Roadmap to integrate HTC, HPC, Data, Instruments, ...– Policy & services needed to run a grid
• Governance & ownership by its stakeholders– EGI Council votes proportional to GDP– EGI Council fees proportional to votes– Sustainable small coordinating organisation (EGI.eu)– Builds on resources from within its stakeholders
• Location selected to be Amsterdam– Distributed staff with a core (~50%) in Amsterdam– Approximately 40 staff, €3.5M/year
Swiss Grid Day 6
EC Funding
National Funding
Relationship betweenNGIs & EGI
Swiss Grid Day 7
NGI National Tasks (NGI)
NGIInternational
Tasks (EC & NGI)
NGI National Tasks (NGI)
NGIInternational
Tasks (EC & NGI)
NGI National Tasks (NGI)
NGIInternational
Tasks (EC & NGI)
EGI.euEGI Global Tasks (EC, NGI, EGI.eu)
EGI means Innovation
• Deploy Technology Innovation– Distributed Computing continues to evolve
• Grids Desktops Virtualisation Clouds ?
• Enable Software Innovation– Provide reliable persistent technology platform
• Today: Tools built on gLite/UNICORE/ARC
• Support Research Innovation– Infrastructure for data driven research
• Support for international research (e.g. ESFRI)
Swiss Grid Day 8
Technology Innovation
• Will come from outside EGI– EGI runs DCI technologies in production
• Partnership with technology projects– EMI (European Middleware Infrastructure)
• Continued evolution of gLite, ARC, UNICORE
– RESERVOIR• Public cloud functionality to ERA
– StratusLab (Proposed)• Use RESERVOIR to support gLite environments
Swiss Grid Day 9
Swiss Grid Day 10
Prototype
Requirements
Implementation
Usage
Release
Evaluate
Deployment
Feedback
Collaboration
Technology innovations taking place outside the production infrastructure
Innovating technology being deployed within the production infrastructure
NGIs & EIROs
EC &National
ResearchProjects
Release
Software Innovation
• Will come from outside EGI– EGI is a neutral platform for applications
• EGI cannot support all services in its core– Every community needs something different
• Foster innovation within ‘sectors’– High Throughput Computing– Digital Libraries– ...
Swiss Grid Day 11
EGEE Maintained Components External Components
Example of Deployed Services
12Physical Resources
General Services
LHC FileCatalogue
HydraWorkload
Management Service
File TransferService
Logging &Book keeping
Service
AMGA
Storage Element
Disk Pool Manager
dCache
Information S
ervices
BDII
MON
User InterfaceUser Access
SecurityServices
Virtual Organisation Membership
Service
Authz. Service
SCAS
Proxy Server
LCAS & LCMAPS
Compute Element
CREAM LCG-CE
gLExec
BLAH
Worker Node
User Interface
BDII
CORE UMD SITE SERVICES (TBC)
COMMUNITY SERVICES
RESPECT USER
Deployed by EGIMaintained & Supported
through EGI
Deployed by EGIMaintained &
Supported by the community
Digital Library Community
• Community specific software stack– gCube from D4Science project
• VREs: Manage access to libraries– Big focus on usability
Swiss Grid Day 13Physical Resources
Core UMD Services
Information Service
Virtual Research
Environment (VRE)
Manager Service Container
VREServices
VREServices
High Throughput Computing
• Managing large collections of jobs & files– HEP, LS, ...
• Problem specific Applications and Tools– Maintained & supported by that community
Swiss Grid Day 14Physical Resources
Core UMD Services
GANGA DIANE
HTC Community Services (e.g. WMS, FTS, LFC , AMGA, ...)
User ApplicationsWISDOM
Research Innovation
• An infrastructure to support the ERA– Within the EU27– Geographical Europe– Interoperability worldwide for collaboration
• Work with Virtual Research Communities– Groupings of aligned Virtual Organisations– Provide community specific:
• Support, training, consultancy, requirements etc.
Swiss Grid Day 15
Swiss Grid Day 16
The ESFRI Projects
• First roadmap 2006 and updated in 2008 with 44 projects
• Preparatory phase funding for most with second round soon
• Big push in FP8?
ESFRI: European Strategy Forum on Research Infrastructures
ESFRI Environmental Sciences
LIFEWATCH
EMSO
IAGOS-ERI
AURORA BOREALIS
EUFAR-COPAL
ICOS
EURO-ARGO
EPOS
EISCAT-3D
SIOS
Status 2009
17Swiss Grid Day
EGI User SupportU
SE
RS VOs
VOs
Virtual Research
Community
UserForum
SteeringCommitteeU
SE
RS VOs
VOs
Virtual Research
Community
US
ER
S
NGI
NGIHelpdesk
EGI.eu
Training Events
Trainers
Apps.DB
EGIHelpdesk
VRCHelpdesk
OtherHelpdesk
ESFRI Project
VOs
Virtual Research
Community
18
Swiss Grid Day 19
Moving from EGEE to EGI
• What is different?– EGEE did ‘everything’– EGI focuses just on infrastructure operations
• What is the same?– Running a 24/7 production quality infrastructure– Providing a support framework for the users– Collaboration to drive European DCI forward
The EGI-InSPIRE Project
Integrated Sustainable Pan-European Infrastructure for Researchers in Europe
• Submitted to INFRA-2010 1.2.1• Targeting call objectives:
– 1.2.1.1: European Grid Initiative– 1.2.1.2: Service deployment for heavy users
• A 4 year project• Seeking a total €25M EC contribution
– Project cost €70M
How much does it cost?
24
25
Annual Running Costs EGEE-III (M€)Staff (EC & National)
Hardware (National)
5127
14
8
% Staff Effort in EGEE-III
Operations User CommunityMiddleware Coordination
Annual CostsEGEE EC Contribution: €16M
(EGI Equivalent: ~€11M)Infrastructure Cost: ~ €80M
EGI Project Ecosystem
EGI
SoftwareProviders
SoftwareProviders
SoftwareProviders
Com
munity
Com
munity
VRC
VRC
SSC
SSC
EC Projects
VRC
SpecialisedSupportCentres
SupportingProjects
Virtual ResearchCommunities
22
Project Structure
23Project Management
External Relations: Policy, Dissemination and Standards
SA3SoftwareSupport
SA2Middleware
Unit
SA1Operations
JRA1Operational
Tools
SA4Services for
HUCs
NA3User
SupportSoftwareProvisioning
andSupport
Operations and Tools
User Support and
Services
Project Structure
EGI-InSPIRE
Managerial Collaborations (e.g. EC, DEISA, OSG, ...)
Technical Collaborations(e.g. CHAIN, GISELR, SIENA, ...)
VRC & SSCProjects
(e.g. ROSCOE, SAFE, CUE, TAPAS, ...)
SoftwareProjects
(e.g. EMI, IGE, SGI...)
User Support & Services
• Support User Communities– Researchers in International Collaborations– Scale up from the single VO to a community
• Provide a federated Helpdesk linking:– Discipline specific support (e.g. Bio Apps)– National infrastructure support (e.g. NGS)– Generic services (e.g. Training)
• Provide core services to support users– Manage VOs, Application DB, Training DB
Swiss Grid Day 25
Possible Virtual Research Communities (FP7 funding)
• High Energy Physics (ROSCOE)• Life Sciences (ROSCOE)• Computational Chemistry & Material Science (ROSCOE)• Grid Observatory [Computer Science] (ROSCOE)• Complexity Science (ROSCOE)• Photon Science (ROSCOE)• Humanities (ROSCOE)• Astronomy & Astrophysics (SAFE)• Fusion (SAFE)• Earth Science (SAFE)• Molecular Biodiversity (SIMBIOME)• NMR in Structural Biology (WeNMR)• Hydro-Meteorology (DRIHME)
Swiss Grid Day 26
Training Events
Registry of Trainers
Applications Database
29
http://appdb.eu-egee.org
Alt. link: http://grid.ct.infn.it/egee_applications/
EGEE RESPECT EGI
• Identify third-party software that works well with Grid– gLite: http://technical.eu-egee.org/index.php?id=290 – Expand to include UNICORE, ARC, Globus, …
• Simplified Access– P-GRADE, Ganga, Migrating Desktop,
g-Eclipse, i2glogin, Virtual Control Room• Workload Management
– GridWay Metascheduler, DIANE• New Resources
– GRelC, Instrument Element• Infrastructure Services
– StoRMSwiss Grid Day 30
Other Activities
• Dissemination– With NGIs, VRCs, SSCs and other projects
• Support for Heavy User Communities– General & community specific services
• Events– Two Annual meetings: Users & Technology
• Technology– Liaison with software providers– Definition and verification of requirements
Swiss Grid Day 31
Resources
• EGI.eu does not own or control resources• Resources owned by individual organisations
– They manage access for their user communities• Middleware enables secure authorised access
for external collaborators– EGI provided Unified Middleware Distribution (UMD)– EGI defined software & interfaces– UMD compatible software must be deployed– Interoperation within your country and internationally
Swiss Grid Day 32
Transition to EGI Model
• Has already started within EGEE-III• Will continue in EGI-InSPIRE
– Integrated Sustainable Pan-European Infrastructure for Researchers in Europe
– 4 year project, seeking €25M EC contribution• User facing changes will be gradual
– Focused communication from EGI & EGEE• Main public points of contact have few changes
– Helpdesk & software
Swiss Grid Day 33
Summary
• Move away from a project based model– Sustainable e-Infrastructure
• User services will evolve to this new model– Coordination of services provided through NGIs– Expect little change in the delivered services
• Start coordinating European e-Infrastructure– 1st May 2010
• Contact: [email protected]
Swiss Grid Day 34