From requirements through objectives to development plans

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

From requirements through objectives to development plans

Balázs Kónya (Lund University)Technical Director

2nd All-Hands meetingLund, 31 May 2011

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

outline

EMI is about software, the mission of the project is to deliver software for DCIs

•What are we supposed to do after EMI 1•Workplan preparation•The high-level view on the Workplan

Without dwelling into the details

https://twiki.cern.ch/twiki/bin/view/EMI/DeliverableDNA132

31/05/2011 Lund All-hands 2

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Congratulation!


Kebnekaise, 12 May 2011

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Next peak to climb


Matterhorn, April 2012

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

How do we get there?


Technical workplan: DNA1.3.2

Area workplans: DJRA1.1.2, DJRA1.2.2, DJRA1.3.2, DJRA1.4.2

Development plan (PT tasks): (tracker)

We need a plan!

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611the plan: software portfolio

EMI products (DNA1.3.2)•58 products of different maturity level•51 products are released as part of Kebnekaise

– from AMGA to WMS

•3 products are under certification – FTS, Hydra, StoRM

•4 products are in its early development – STS, Pseudonimity, EMI Service Registry,

EMI Messaging

•5 items are already dropped– Grid Manager, SCAS, SLCS, gstat-validation,

update-crls

•Couple of things are missing– EMI_datalib, EMI_auth, ...


EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611the plan: product portfolio


EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

the plan: requirements

• EMI listens to its customers • Requirements of different kind were comunicated via

direct email, verbally, EGI-TCB• PTB assesses and prioritizes requirements

– Adjusts and/or creates new objectives– Continous process, though requirements don’t have an

immediate effect on workplan

• Everything is recorded and tracked:– https://savannah.cern.ch/task/?group=emi-req

• Note: initial set of requirements already were taken into account by DoW and DNA1.3.1– These are not in the tracker


EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

the plan: top priority requirements• WMS should provide better feedback about jobs, automated

resubmission of stuck jobs• Release EMI clients on SL5/32• Standard file locations for logs and temporary files• Publish mw service version• Provide Nagios probes for EMI services• Better error messages (more verbose, informative)• Coherency of command line commands parameters• Coherency of APIs• Debian 6 support• Ubuntu support at least for EMI clients• CEs should support a set of LRMS• Reduce memory consumption of CREAM• Support for SL6


EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

the plan: not accepted requirements

• Fixing known bugs before adding new features

• Uniform logging• Improving (automatization) yaim

configuration• Unicore GUI improvements


EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

High-level Roadmap• Phase-1 (completed with Kebnekaise, see Morris’ JRA1 slide):

– Software integration: created a distribution– Agreements: EMI-ES, STAR, GSI-replacement, messaging use cases – Design: EMI_authlib, EMI Registry– Completed tasks: server-side GLUE2, consistent SRM, ...

• Phase-2 (Matterhorn)– Functionality integration– Implementation of all the agreements– Furher Agreements: compute accounting, delegation, ...– Delivery of the Consolidation plans– Design: AAI strategy, Cloud strategy

• Phase-3 (Monte Bianco)– Realization of the consolidation plans– Implementation of AAI and Cloud strategy– Finalization of developments by bringing code to production level– New platforms


EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

The plan: objectivesCompute area objectives Priority Due

C1 Glue 2.0 support in job management services (LDAP and/or XML rendering). 5 M16

C2 Agreements over an EMI compute accounting record (UR). 3 M16

C3 Investigate solutions to improve interactive access capability of at least one EMI CE. 2 M18

C4Proposal for a common parallel execution framework, a “backend” across the different computing services to allow users to execute parallel applications in a uniform way.

3 M18

C5Implementation of the agreed common job submission and management methods (EMI-ES interface) in all the CEs.

5 M19

C6Implementation of the agreed common job submission and management methods (EMI-ES interface) in compute clients.

5 M20

C7EMI compute area services should provide fully integrated solutions to interface with identified set of batch systems.

4 M20

C8 Glue 2.0 support in matchmaking modules and client tools. 5 M22

C9 Support for the EMI compute accounting record (UR) by compute area services. 3 M22

C10 Consolidation and harmonization of compute area clients/APIs. 4 M25

C11 Implement the EMI cloud strategy within compute area. 3 M32

C12Implementation of the common parallel execution framework across the EMI computing services.

3 M32

C13Extend the parallel computing capabilities to better address multi-core jobs on emerging architecture resources; multi-node execution on interconnected clusters; and special scenarios like advanced topologies, FPGAs, GPGPUs.

3 M32


EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

The plan: objectivesData area objectives Priority Due

D1 A storage client is capable consuming GLUE 2.0 information published by storage elements. 5 M16

D2Investigate the possibility to support http/webdav for LFC in order to provide a standard user-friendly access method to catalogues.

3 M16

D3SRM-capable clients and services should add “file://” to the already supported access protocols.

3 M18

D4 File Catalogue Access from UNICORE data. 2 M18

D5Providing a common set of data access libraries at least between gLite and ARC (EMI_datalib).

4 M22

D6All storage elements publishing full set of GLUE 2.0 storage information and the EMI data client is capable consuming that.

4 M22

D7 Integration of SRM-based access into UNICORE storage management 4 M22

D8 All storage elements offering support for the http(s) protocol. 3 M22

D9 Investigate solutions to work with EMI data services in the context of persistent data Ids. 2 M24

D10 Solve the synchronization problem of the storage elements and the file catalogue. 5 M25

D11 Storage elements offering support for the WebDav protocol. 3 M25

D12Using https instead of httpg for the SRM protocol as a production implementation in all the storage elements and clients utilizing the EMI delegation.

3 M25

D13 Integration of AMGA-based access to UNICORE storage management. 3 M28

D14Design and implement the next generation FTS, a distributed next generation file transfer service that amog others utilizes the common messaging system.

3 M30

D15 Completed migration to the common EMI_datalib data access libraries. 4 M32

D16 Add support for storage space usage accounting to SEs/FTS based on the agreed record. 3 M32

D17 Implement the EMI cloud strategy within data area. 3 M32


EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

The plan: objectives

Security area objectives Priority Due

S1 Plan for substantial simplification and reduction in the number of security area CLIs, libraries, internal components and services: security area consolidation plan. 5 M18

S2

Simplified management of security credentials by reducing the complexity of handling certificates and integrating different security mechanisms like Shibboleth and Kerberos across the EMI stack that allows users to use their own authentication system to access a ``Grid''.

5 M22

S3 Provide common authentication libraries supporting X.509 and optionally SAML (EMI_authlib). 4 M22

S4 Agreement and full support for a common single X.509 and SAML based Attribute Authority Service integrated with all EMI components. 3 M25

S5 Implementation of the security area consolidation plan. 3 M32

S6 Provide a transparent solution for encrypted storage utilizing ordinary EMI SEs. 3 M32


EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

The plan: objectivesInfrastructure area objectives Priority Due

I1Investigate possible use cases for a common standard messaging system in the accounting area.

4 M14

I2Investigate possible use cases for a common standard messaging system for the service monitoring and management.

4 M14

I3Devise a plan for substantial simplification and reduction in the number of infrastructure area CLIs, libraries, internal components and services.

4 M18

I4 Deliver the EMI cloud architecture and strategy. 3 M18

I5Fully utilize and support the GLUE2 model in information components including the development of validation tools.

5 M20

I6 Implement the EMI Service Registry. 5 M22

I7Provide guidelines for 3rd parties to integrate messaging into their services or applications based on the EMI experience.

3 M24

I8

Investigate service remote management interface for the compute, data, security and infrastructure area services, including remote configuration change and service management, utilizing the messaging system.

3 24

I9Implement or adapt the accounting record publishers of compute and data area services to use the common messaging system.

4 M28

I10 Implementation of the infrastructure area consolidation plan. 3 M30


EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

The plan: objectivesCommon or Cross area objectives Priority Due

X1Publish coherent GLUE2-based version information as part of service description in order to facilitate service discovery and monitoring.

4 M16

X2 Definition of a common SAML profile all over the middleware stacks. 3 M16

X3 Agreement on common EMI delegation method. 5 M18

X4 Provide and support monitoring probes for EMI services (e.g. Nagios). 5 M18

X5Integration of the compute area services with the ARGUS authorization framework.

4 M18

X6Adhere to operating system standards for service operation and control regarding configuration, log and temporary file location and service start/status/stop.

4 M18

X7Port, release and support EMI components on identified platforms (full distribution on SL6 and Debian 6, UI on SL5/32 and latest Ubuntu).

4 M22

X8Improve usability of client tools based on customer feedback by ensuring a ) better more informative, less contradictory error messages b) coherency of commands line parameters.

4 M22

X9Initial integration of the storage elements with the ARGUS authorization framework.

4 M22

X10Introduce minimal DOS protection for EMI services via configurable resource limits.

3 M28

X11 Implementation of the EMI SAML profile all over the middleware stack. 3 M28

X12The legacy Globus security infrastructure (GSI) will be replaced with a common security solution based on TLS/SSL and EMI delegation method.

4 M30

X13Provide optimized semi-automated configuration of service backends (e.g. databases) for standard deployment.

3 M30

X14Adapt or implement monitoring interfaces, sensors, providers for compute, data, security and infrastructure services to allow the use of standard monitoring tools preferably based on the common EMI messaging system.

2 M30

X15 Increase performance of EMI services. 4 M32

X16 Evolve EMI components to meet specific user requests. 3 M32

X17 Complete the rewrite of components utilizing the new emi_authlib libraries. 3 M32


EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Development tasks

• Comming soon:

• Partially covered by the area workplans

• https://savannah.cern.ch/task/?group=emi-dev


EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Thank you

31/05/2011 18Lund All-hands

EMI is partially funded by the European Commission under Grant Agreement INFSO-RI-261611

Documents

From requirements through objectives to development plans