Reliable File Transfer:Lessons Learned

Bill Allcock, ANL

Ravi Madduri, ANL

Overview Quick Overview of the Service Design Issues

Lifetime Management Virtualization Service Data Elements

Implementation Issues SOAP Processing Issues Standardization Avoid Language specific Types Anyhelper API Fault Tolerance

RFT in Action

Registry

* The scenarios in this presentation are offered as examples and are not prescriptive

1. A Grid Service Container is started up; It contains an RFT Factory service; The RFT Factory service registers itself

RFT Factory

Grid Service Container

Client

RFT in Action

Registry

* The scenarios in this presentation are offered as examples and are not prescriptive

2. From a knownregistry, the client discovers a

factoryby querying theService data of theregistry

RFT Factory

Grid Service Container

Client

RFT in Action

3. The client calls thecreateServiceoperation on the factory and passes in a TransferRequest

RFT Factory

Grid Service Container

* The scenarios in this presentation are offered as examples and are not prescriptive

Client

RFT in Action

RFT Factory

Grid Service Container

RFT Service Instance- Start the Instance- Deserialize XML to Java- Write Request via JDBC- Persist Service State

4. The instance is started, and the factory returns a locater

* The scenarios in this presentation are offered as examples and are not prescriptive

Client

RFT in Action

RFT Factory

Grid Service Container

RFT Service Instance- Start the Instance- Deserialize XML to Java- Write Request via JDBC- Persist Service State

5. Client calls Start(), subscribes to notificaitons, etc.

* The scenarios in this presentation are offered as examples and are not prescriptive

RFT in Action Service is OGSI

compliant Uses existing

GridFTP (non-OGSI) protocols and tools to execute 3rd Party Transfer for the user

Provides extensive state transition notification

GridFTPServer

GridFTPServer

RFT ServiceInstance

* The scenarios in this presentation are offered as examples and are not prescriptive

Lifetime Management

Lifetime management is a key aspect of OGSI

Was not intuitively clear how to handle this for “disconnected” services

Our (perhaps not optimal) solution is to give it an indefinite lifetime

Should there be an activity monitor? Does that really solve the problem? Any other ideas?

Virtualization Another Key Design Issue in services We virtualize data movement

It works, LBL and ANL have interoperable implementation

Need to standardize… Data Services Virtualization from DAIS

Should we pass around GSHs of file services rather than URLs?

Granularity Single file .vs. Multi-file .vs. service

composition

Service Data Elements A huge improvement over the non-OGSI

services Information Services are (should be) baked

in to the services Defined both push (event notification) and

pull (full transfer status) SDEs Need to be cognizant of size, frequency,

and performance of notifications Lots of interesting possibilities: bandwidth,

errors, network status, etc..

Implementation Issues

SOAP Deserialization Deserialization can be a HUGE issue. Our

original (very simple) XML could take up to 30 minutes to process.

A straight forward change reduced that drastically.

SOAP engine also needs to be looked at. Standardization

Critical for success Will be moving to the OGSI-Agreement

interface

Implementation Issues (cont.)

Language Specific Data Types Avoid them (I.e. Java vector type) A python based service would not be able to

deserialize that. AnyHelper API

Will deserialize any general XML blob that utilizes basic types.

Implementation Issues (cont.)

Fault Tolerance Multiple levels of Fault tolerance GridFTP will handle remote failures RFT provides fault tolerance of request via

JDBC compliant database Service container provides instance fault

tolerance We write only the primary key into the wsdd

file to avoid slowing down container restart.