CHEP 2003 Iosif Legrand DIAMONDS DIstributed Agents for MObile & Dynamic Services Iosif C. Legrand California Institute of Technology

CHEP 2003 Iosif Legrand

DIAMONDS DIstributed Agents for MObile & Dynamic

Services

Iosif C. LegrandIosif C. Legrand

California Institute of Technology


This project was developed in collaboration with This project was developed in collaboration with

a team from NUST, Pakistan :a team from NUST, Pakistan :

Saad KianiSaad Kiani

Maria RiazMaria Riaz

Aamir ShafiAamir Shafi

Anjum ShehzadAnjum Shehzad

Umer FarooqUmer Farooq

coordinated by : Arshad Alicoordinated by : Arshad Ali


Distributed Service Architecture Distributed Service Architecture Design Considerations Design Considerations

The prototype for Distributed Dynamic Services we are developing gather, disseminate and coordinate time-dependent information and services for specific HEP applications.

It aims to extend and improve the (centralized) client – server architectures on large scale distributed systems.

Hierarchical structure of loosely coupled services which are independent & autonomous entities able to cooperate using a dynamic set of proxies or self describing protocols.

These services should facilitate an efficient use of the distributed resources and provide adaptability.

Reliable on a large scale network distributed environment

Avoid single points of failure

Automatic re-activation of components and services

Scalable & Flexible for adding dynamically new services and automatically replicate existing ones to cope with time dependent load


JAVA RMI & JINI JAVA RMI & JINI Distributed Object Systems Distributed Object Systems

LookupServiceStubLookup

Service Skeleton CLIENTServer

“Traditional” Distributed Object Models(CORBA, DCOM)

“IDL” Compiler

Dynamic Code Loading Less Protocols !!

LookupServiceStub

(Proxy)CLIENT

LookupServiceStub

(Proxy)

Service

The Stub is linked to the Client.The Client must know about theservice from the beginning and needs the right stub for it

Any service can be used dynamically


Jini & Code MobilityJini & Code Mobility

The purpose of the Jini architecture is to federate groups of devices and soft-ware components into a single, dynamic distributed system.

Jini is a set of specifications that enables services to find each other on a network and allows these services to participate within certain types of operations within the framework. It allows these services to interact in a dynamic and robust way: no user intervention when devices are added or removed, adaptability when services come and go, no prior knowledge of the services implementation are needed simplifying administrative duties.

It provides support for a service oriented architecture with code mobility

JINI extends the Java Platform providing support for Distributed Applications


JINI OverviewJINI Overview

The purpose of the Jini architecture is to federate groups of devices and soft-ware components into a single, dynamic distributed system.

Lookup Service

Services are found and resolved by a lookup service. The lookup service is the central bootstrapping mechanism for the system and provides the major point of contact between the system and users of the system.

Leasing

Access to many of the services in the Jini system environment is lease based. A lease is a grant of guaranteed access over a time period. Each lease is negotiated between the user of the service and the provider of the service as part of the service protocol.

Remote Events

The Jini architecture supports distributed events. An object may allow other objects to register interest in events in the object and receive a notification of the occurrence of such an event. This enables distributed event-based programs to be written with a variety of reliability and scalability guarantees.


JINI Overview(2)JINI Overview(2)

Transactions Manager

A series of operations, either within a single service or spanning multiple ser-vices, can be wrapped in a transaction. The Jini Transaction interfaces supply a service protocol needed to coordinate a two-phase commit.

The JavaSpaces Service

which can be used for simple communication and for storage of related groups of objects written in the Java programming language

The Mailbox Service

Can be used to provide asynchronous communications (based on any type of messages ) between distributed services

Security

The design of the security model for Jini technology is built on the twin notions of a principal and an access control list. Jini services are accessed on behalf of some entity which generally traces back to a particular user of the system. Services themselves may request access to other services based on the identity of the object that implements the service.


Lookup Discovery Service Lookup Discovery Service

A Service Registers with at least one Lookup Service using the same ID.

It provides information about its functionality and the URL addressed from where interested clients may get the dynamic code to use it.

The Service must ask each Lookup Service for a lease and periodically renew it.

If a Service fails to renew the lease, it is removed form the Lookup Service Directory. When problems are solved, it can re-register.

The lease mechanism allows the Lookup Service to keep an up to date directory of services and correctly handle network problems. Discovery mechanism implementation can be enhanced with UDDI , JXTA protocols.Discovery mechanism implementation can be enhanced with UDDI , JXTA protocols.

Service

LookupServiceLookupService


CLIENT

Register

Service IDRegister with ID

Ask for a leaseGet a lease for T

jar

Web Server

Publish the“Interface” jar

jar

Web Server



Remote EventsRemote Events

Any Interested Service or Client in certain types of events must register. It must ask for an event notification lease and periodically renew it.

If the Event Provider or the Event Listener fails the system will correctly identify such a state as well as the case when such a unit becomes again available.

The lease mechanism provides for any Event Producer a real list of active Listeners. Any Event Consumer is informed if the Producer is alive .

Service


CLIENT

Register for Event Notification

Ask for a leaseGet a lease for T

ServiceEvent Provider


JavaSpacesJavaSpaces

JavaSpace ->Tuplespace computation model (LINDA) & Distributed Object Model

Loosely coupled/anonymous communication. Participants are no aware of each others identity and the message objects are selected and retrieved using associative lookup based on pattern matching

Persistence of the Tuple Space does not require that participants exist concurrently (timely uncoupled)

Allows Code and Data Migration (tuples=code & data)

Well suited for many parallel algorithms in dynamic configurations.

Process

Process

Process Tuplespace

Take Write


Transaction Services Transaction Services

Transactions enable operational grouping into all successful or all failed. It allows a two phase commit operations in a distributed environment.

A Client hands a factory method to a Transactions Manager reference and a lease duration and receives a Transaction Object and a Lease.

The Client pass the Transaction Object to the Services it wants to use asking to do the task “under the transaction”.

If the Client or any participant aborts the transaction the Transaction Manager instruct all the participates to “rollback”.

If the Client asks to commit the transaction, the Transaction Manager queries all the participants and if all are “prepared” it asks all of them for “ roll forward” and commit the transaction.


The System FunctionallyThe System Functionally

This prototype system, based on Jini, aims to provide:

Customized dynamic services

“Smart Proxies”

“Mobile Agents”

These conceptual entities should be able to work together and interact in a distributed environment by:

Remote Event Subscription/ NotificationSynchronous and asynchronous communicationCode Mobility Persistent Objects


Dynamic Services

The Station Servers allows to dynamical load into the system local services. This procedure may be initiated by a user or by an other service.

The Station Servers register the dynamic services with the lookup services, test them and continuously keep them available by renewing the leases.

The additional “mobile code”, partners may need to use a service is either loaded on a http server or on a JavaSapace.

StationServer

Service Discovery

StationServer


Dynamic Service

Dynamic Service

ClientClient

Registration

jar

Web Server


jar

Web Server


Dynamic Service

Dynamic Service

Tuplespace


Dynamic Proxies

Smart proxies may :

• Dynamically load part of the service code to the client to improve performance.

• Cache locally frequently used data from the remote service and keep them synchronized

• Discover and select the most convenient places to access common services from the “client” perspective.

LookupServiceProxy

CLIENTStationServer Lookup

ServiceService

“Smart” Proxies extend the functionality of the network services by trying to provide an optimized performance based on the locally available resources, services or data.


Mobile Agents

Mobile Agents are automatons entities able to migrate between Station Servers.

Each Station Server offers the runtime environment, priority and controls the access permission for the hosted agents.

Mobile Agents interact with the Station Servers network to get access to the necessary information to migrate and to perform a certain task.

The agent migration is done as a weak migration ( e.g. the agent prepare itself before it is moved and save its state into structured which a serialized)

Mobile Agents should remain small and simple entities. To achieve this, and provide complex functionality they must use and access “static” services hosted by the Station Server network.


Mobile Agent Framework DevelopmentMobile Agent Framework DevelopmentDIAMONDSDIAMONDS

Develop a framework that allows for hosting dynamic services in the Develop a framework that allows for hosting dynamic services in the form of Mobile Agentsform of Mobile Agents

Allows for registration/publishing and discovery of servicesAllows for registration/publishing and discovery of services Support for remote event notification Support for remote event notification Trust Establishment between servicesTrust Establishment between services Enabling access to remote resources through AgentsEnabling access to remote resources through Agents Reliability over a large distributed networkReliability over a large distributed network

Avoid loss/duplication of agents – transactions Avoid loss/duplication of agents – transactions Dynamic Interaction with other services toDynamic Interaction with other services to

Use other service’s generated dataUse other service’s generated data Help another service in decision makingHelp another service in decision making


Mobile Agents and Code MobilityMobile Agents and Code Mobility

Object moves with its code definitionObject moves with its code definitionAgents are packed in Jar files which serve as the ‘codebase’ for Agents are packed in Jar files which serve as the ‘codebase’ for

the agentsthe agents Marshalled Object for the agent is created which stores the Marshalled Object for the agent is created which stores the

codebase information.codebase information. On un-marshalling, the agent’s definition is requested from its On un-marshalling, the agent’s definition is requested from its

codebase.codebase.Weak MobilityWeak Mobility

Code and Variable state is transferred.Code and Variable state is transferred.

Dynamic Services register their proxies with the discovery services.Dynamic Services register their proxies with the discovery services. Client interested in using these services needs to download the proxyClient interested in using these services needs to download the proxy Client need not be aware of the actual location of the service. Client need not be aware of the actual location of the service. Service’s proxy communicates back to its service for the client.Service’s proxy communicates back to its service for the client.

Redundant codebases for downloading proxy definitionRedundant codebases for downloading proxy definition

““Proxy” is the entire service Proxy” is the entire service


Dynamic Services in this framework may have a GUI Dynamic Services in this framework may have a GUI component (act as complex proxy) that is created at runtime, component (act as complex proxy) that is created at runtime, on request by a client.on request by a client.

These user interfaces are transferred to the client and These user interfaces are transferred to the client and opened at the Client sideopened at the Client side

Clients can perform operations on the services, remotely, Clients can perform operations on the services, remotely, through their User Interfaces.through their User Interfaces.

Each client has a unique state for the GUI depending on the Each client has a unique state for the GUI depending on the operations it is performing.operations it is performing.

When a service un-registers, its remote GUI is closed and on When a service un-registers, its remote GUI is closed and on re-registration, it is re-opened automatically at the clients re-registration, it is re-opened automatically at the clients that were previously viewing the GUI.that were previously viewing the GUI.

Remote User Interfaces for Dynamic ServicesRemote User Interfaces for Dynamic Services


Transaction Service controls Mobile AgentsTransaction Service controls Mobile Agents

Transactions are used in the framework wherever atomic operations Transactions are used in the framework wherever atomic operations are requiredare required

Agent Movement from one Station to another needs to be atomicAgent Movement from one Station to another needs to be atomic (Un) registration(Un) registration Code and state transfer / Migration Code and state transfer / Migration Re registrationRe registration

Either the move operation is carried out completely and successfully, Either the move operation is carried out completely and successfully, otherwise, it stays where it was.otherwise, it stays where it was. Avoid agents loss and duplicationAvoid agents loss and duplication

Implemented using JINI Transaction ServiceImplemented using JINI Transaction Service


SecuritySecurity

Asymmetric keys and CertificatesAsymmetric keys and Certificates Each service signed with owner/administrator’sEach service signed with owner/administrator’s

private keyprivate key Hosting Services (Stations) maintain a trust store Hosting Services (Stations) maintain a trust store

filled with trusted certificatesfilled with trusted certificates

Trust between AgentStation and Agent’s OwnerTrust between AgentStation and Agent’s Owner Key StoresKey Stores

Signing Jars, Generating CertificatesSigning Jars, Generating Certificates Trust StoresTrust Stores

Trusted CertificatesTrusted Certificates Import, Send, Delete CertificatesImport, Send, Delete Certificates

Certificate verification on loading and moveCertificate verification on loading and move

Trust between Agent and Remote ClientTrust between Agent and Remote Client Open Agent – Remotely accessible To All ClientsOpen Agent – Remotely accessible To All Clients Close Agent – Remotely accessible To Owner OnlyClose Agent – Remotely accessible To Owner Only

Agent Verifies if the requesting client is its true ownerAgent Verifies if the requesting client is its true owner

Agent Station

PrivateKeystoreTrust

Store


Agent StationAgent Station

An Agent Station is a network service that can host Dynamic Services.Station Servers are dynamically interconnected (peer-to-peer) providinga distributed frame for Services.

Handling Remote events

Keeps a dynamic list of available services

Performs security authentification / encrypted communication

Support for Inter-Service communications (Roaming for mobile Agents)

StationServer

StationServer

StationServer

LookupService

LookupService

Proxy Exchange

Registration

Service ListenerRemote Notification


Agent StationsAgent Stations

Agent Stations are deployed at various Agent Stations are deployed at various locations across the network (LAN/WAN) locations across the network (LAN/WAN)

Agents can be loaded by a user.Agents can be loaded by a user.

The agent can move to other stations.The agent can move to other stations.

Where ever the agent goes, it always registers Where ever the agent goes, it always registers itself as a service, making remote itself as a service, making remote administration/monitoring possible for the administration/monitoring possible for the user user

Agent may be open or closedAgent may be open or closed Open – Anyone can view what the Open – Anyone can view what the

agent is doing i.e.it’s a free serviceagent is doing i.e.it’s a free service Closed – The activities of this agent Closed – The activities of this agent

can only be monitored and results can only be monitored and results retrieved by its owner.retrieved by its owner.

The agent mobility and its monitoring are all The agent mobility and its monitoring are all subject to its security credentialssubject to its security credentials

A

B

Registration Service

Agent Station B

Agent Station A

Agent

Agent Proxy

Station Proxy


Mobile Agents ServicesMobile Agents Services

Agents are Dynamic & Mobile ServicesAgents are Dynamic & Mobile Services Generic Agent with mobility mechanism to carry the Generic Agent with mobility mechanism to carry the softsoft

statestate Service Agents with registration mechanism & remote UI, Service Agents with registration mechanism & remote UI,

extended from Generic Agentsextended from Generic Agents

Basic Application AgentsBasic Application Agents Test Service AgentTest Service Agent Database Access AgentDatabase Access Agent File System Access AgentFile System Access Agent Search agent Search agent

The Basic Application Agents should be extended for specific The Basic Application Agents should be extended for specific applications applications


Database Access AgentDatabase Access Agent

Provides access to databases nearby the host AgentStation.Provides access to databases nearby the host AgentStation. MySQLMySQL MS-AccessMS-Access RmiJDBCRmiJDBC

Host AgentStation keeps record of the dbs on the LAN.Host AgentStation keeps record of the dbs on the LAN.

Load and send the Agent to an AgentStation and then access the Load and send the Agent to an AgentStation and then access the databases there remotely through the Agent’s UI.databases there remotely through the Agent’s UI.

Remote DB BrowserRemote DB Browser


Remote File Access AgentRemote File Access Agent

Provides access to the host Agent Station's File System Provides access to the host Agent Station's File System depending on security parametersdepending on security parameters View, Edit filesView, Edit files Remote File BrowserRemote File Browser


Read Performance of Agent

0

1

2

3

4

5

6

7

8

9

10

1.95 4.88 7.81 11.72 14.65 17.58

File Size (MB)

Sp

ee

d (

MB

/se

c)

Local Agent

LAN(100Mbps)

LAN(10Mbps)


Write Performance of Agent

0

1

2

3

4

5

6

7

8

1.95 4.88 7.81 11.72 14.65 17.58

File Size (MB)

Tra

ns

fer

Ra

te(M

B/s

ec

)

Local Agent

LAN(100Mbps)

LAN(10Mbps)


SummarySummary

A prototype system, based on Java & Jini is currently under development to provide a distributed mobile agents service system to help the physics data analysis in a multi-tier Regional Centre architecture.

Code mobility paradigm provides the mechanism for a consistent, Code mobility paradigm provides the mechanism for a consistent, correct invocation of components in large, distributed systems.correct invocation of components in large, distributed systems.

These services need to use Monitoring information services (MonaLisa) in decision making and the strategy they use to adapted to available resources.

This prototype aims prove the advantages of such an approach compared with a client-server model for handling dynamic information in a large scale distributed system

Documents

CHEP 2003 Iosif Legrand DIAMONDS DIstributed Agents for MObile & Dynamic Services Iosif C. Legrand California Institute of Technology