Facilitating resource discovery in grid environmentswith peer-to-peer structured tuple spaces

Maozhen Li & Man Qi

Received: 9 October 2008 /Accepted: 16 February 2009 /Published online: 4 March 2009# Springer Science + Business Media, LLC 2009

Abstract The computational grid is rapidly evolving into alarge-scale computing infrastructure that facilitates resourcesharing and problem solving over the Internet. Informationservices play a crucial role in grid environments fordiscovery of resources. The dynamic nature and the large-scale of a grid pose many challenges to informationservices in terms of scalability and resilience. This paperpresents RDSpace which can be used as a substrate forresource discovery in grid environments. RDSpace builds ashared tuple space on top of a structured peer-to-peeroverlay to achieve high scalability in dealing with a largenumber of computing nodes and to support range queries indiscovery of resources. Another novelty of RDSpace lies inits capability to handle churn situations where nodes mayjoin or leave the space frequently. RDSpace is evaluatedfrom the aspects of scalability and churn handling, and theevaluation results are also presented in this paper.

Keywords Grid computing . Information services .

Resource discovery . Structured P2P. Tuple space

1 Introduction

The past few years have witnessed a rapid development ofgrid computing systems and applications. Information

services play a crucial role in grid computing environmentsfor resource discovery. Grid middleware technologiesfacilitate information services. For example, the currentGlobus Toolkit1 provides a component called the Monitor-ing and Discovery System version 4 (MDS4) [1] forresource registration and discovery. The MDS4 componentadopts a hierarchical tree structure to distribute its moni-toring data on resources across a virtual organization (VO),in which every node runs an index service monitoring itsresources and pushing this information up to a master indexserver. A query to the top Index server could retrieve all theinformation on the resources available in a VO. TheRelational Grid Monitoring Architecture (R-GMA) [2],which is now a component of the gLite2 middleware, alsofacilitates resource registration and discovery. It is worthnoting that grids differentiate themselves from traditionaldistributed systems in the following aspects:

& The size of a grid is usually large in terms of thenumber of computing nodes involved.

& Resources in a grid are usually heterogeneous withvarious computing capabilities and services.

& A grid is dynamic in that computing nodes may join orleave a grid freely. In addition, some resources such asthe CPU load of a grid node may change frequently.

The aforementioned characteristics of grids bring forth anumber of challenges to existing information services,notably the MDS4 and the R-GMA. The hierarchicalstructure along with centralized management of MDS4has an inherent delay associated with it which potentiallylimits its scalability in resource registration. It might take along time for information on resources to be updated from

Peer-to-Peer Netw Appl (2009) 2:283–297DOI 10.1007/s12083-009-0036-8

2 http://www.cern.ch/glite

1 http://www.globus.org

M. Li (*)Electronic and Computer Engineering,School of Engineering and Design, Brunel University,Uxbridge UB8 3PH, UKe-mail: Maozhen.Li@brunel.ac.uk

M. QiDepartment of Computing, Canterbury Christ Church University,Canterbury, Kent CT1 1QU, UKe-mail: mq4@canterbury.ac.uk

the leaf nodes to the root index service node. Cai et al. [3]point out that the scheme to partition resource informationon index servers is typically predefined and cannot adapt tothe dynamic changes of VOs. The MDS4 also lacks amechanism to deal with failures of index servers which maybreak the information service network into isolated subnets.The R-GMA contains a centralized registry [4], and performspoorly when dealing with only 100 consumer nodes [5].

In parallel development with grid computing, peer-to-peer (P2P) computing is emerged into another promisingcomputing paradigm that typically facilitates file sharing inlarge network environments [6]. P2P networks usuallyorganize peer nodes in a decentralized way, and thereliability can be enhanced by replication of shared filesamong peer nodes. Files can be arbitrarily distributed intopeer nodes without a structure, or they are distributedfollowing a structure such as Distributed Hash Table(DHT). DHT based P2P networks such as Chord [7],Pastry [8], CAN [9] have shown enhanced scalability inrouting lookup messages for files with a guaranteed numberof hops. Foster et al. [10] analyze the differences betweenP2P and grid computing and discussed a possible conver-gence of the two computing paradigms. Talia et al. [11]point out the benefits that P2P networks could bring to gridsystems in terms of scalability and robustness. However,directly applying DHT technologies to grid informationservices mainly poses two challenges. On the one hand,DHT systems usually incur high maintenance overhead indealing with churn situations where peer nodes may join orleave P2P networks at high rates [12]. On the other hand,DHT based P2P networks only support exact matches forfiles using single hash keys. In a grid environment, it is notrealistic to employ a single hash key for a resource whichmay have a number of attributes such as its CPU load,memory space, storage space, and availability. Moreover,the values of these resource attributes dynamically changein a grid environment. Range queries on resources shouldbe supported in grid information services.

It should be pointed out that a few DHT based gridinformation systems have been proposed to support rangequeries in grid information services. For example, SWORD[13] supports multi-attribute range queries to locate suitablepeer nodes. In SWORD, each DHT is used to manage aresource attribute. A query is routed by identifying itspreferred attribute and using the corresponding DHT.Similarly, Building on Chord, MAAN [3] also uses multipleDHTs to manage a range of queries. Mercury [14] isanother example of a grid information service that supportsrange queries over multi-attributes. Like SWORD andMAAN, Mercury maintains a separate logical overlay foreach attribute albeit it does not employ DHTs. Maintaininga DHT or logical overlay for each resource attributeinvolves updating each of them when a resource advertise-

ment is received. As the number of attributes of resourcesincreases, the update traffic would also increase heavily. Inaddition, the heterogeneity of gird resources adds morecomplexity to DHT structured grid information systems.

The Linda model [15] is an approach that facilitates thecommunication and coordination of distributed systems.The original Linda model realises a logical, contentaddressable, associative memory called tuple space that isespecially suited for parallel and distributed computingenvironments. A tuple space represents a repository, whichis able to store, read and remove tuples. The main advantageof a tuple space is the global shared memory abstraction thatprovides a common communication infrastructure accessibleby all participating processes. As the tuple space can beaddressed like an associative memory, the illusion of asystem global hash table is provided to the clients. Theconcrete identity of consumers and producers is hidden, thusallowing all participating components to communicatetransparently. Tuples are stored persistently in the tuplespace, hence the communication between producers andconsumers is decoupled both temporally and spatially. Atuple space can be accessed with simple but powerful storageand retrieval operations that allow matching resourcerequests of consumers. A tuple space simplifies the processof publishing and retrieving resources as it provides a high-level data centric abstraction of an arbitrarily complex system

This paper presents RDSpace, a P2P structured tuplespace model which can be used as a substrate for resourcediscovery in grid computing environments. RDSpace buildson a DHT structured P2P overlay for high scalability indealing with a large number of computing nodes. Tuplesprovide a flexible way in describing resources whichfacilitates range queries in resource discovery. In addition,RDSpace is capable of handling churn situations. Experi-mental results show the effectiveness of the RDSpace inresource discovery.

The remainder of the paper is organized as follows.Section 2 describes in detail the design and implementationof RDSpace. Section 3 evaluates RDSpace from the aspectsof scalability, event latency and churn handling. Section 4discusses related work. Section 5 concludes the paper andpoints out some future work.

2 RDSpace design and implementation

The basic tuple space interface defined in Linda describesfour basic operations:

& out(tuple): Inserts a tuple into the tuple space.& in(pattern): Removes a tuple matching a pattern from

the tuple space. If no matching tuple is available, thisoperation is blocked.

284 Peer-to-Peer Netw Appl (2009) 2:283–297

& read(pattern): Reads a tuple matching a pattern fromthe tuple space, but does not remove it. This operationis blocked if no matching tuple is available.

& eval(expression): Creates an active tuple, which startsan process used for parallel evaluation of the arguments.The resulting tuple is inserted into the tuple space.

These basic operations are sufficient to allow simple butpowerful matching capabilities between consumers andproviders in a distributed system. These operations build ahigh level abstraction on top of a complex system such as alarge-scale grid system, hence simplifying the process ofresource discovery.

2.1 RDSpace architecture

Figure 1 shows the architecture of RDSpace. The compo-nents of RDSpace are described below.

Tuple space abstraction A tuple space provides a datacentric view on a complex and dynamic grid-computingenvironment. This layer provides an abstract tuple spaceinterface to applications, allowing applications to concen-trate on the resources they require instead of the way to findthem.

Tuple space to P2P adaptation This layer is responsible foradapting the abstract tuple space interface to the data centricpart of the P2P infrastructure. This layer defines the tupleformats used by RDSpace.

Data store/lookup P2P overlay network This layer repre-sents the data centric part of a P2P infrastructure. Itsresponsibility is to store and retrieve data on the P2Poverlay network.

Self-organizing P2P routing overlay The main responsibil-ity of this layer is to employ the overlay network protocol

defined by the upper layer onto the underlying physicalnetwork. This layer is responsible for routing messages aswell as handling joining and leaving peer nodes.

2.2 RDSpace on structure P2p overlays

As a tuple space serves as an interface to the P2Pinfrastructure, an adaptation to the underlying P2P networkis required. Table 1 shows how structured and unstructuredP2P networks support the interface required by RDSpace.

A tuple space offers a high level interface to a globalunstructured memory abstraction. It is not important forapplications to know where information is stored or whereit is retrieved from, as a tuple space represents a uniformstorage location. Structured P2P overlay networks offeroperations that are comparable to operations offered by atuple space interface. In contrast to tuple spaces structuredP2P overlays employ a structured view on the physicalnetwork. When a resource provider publishes a resourcedescription, it is stored at a peer node defined by theoverlay routing algorithm. Hence a node different from theprovider may store the resource description. Other peernodes wishing to retrieve the resource description have toaccess that node in order to determine the actual resourceprovider. Unstructured P2P overlays do not employ anystructure on physical networks. The basic lookup mecha-nism in unstructured P2P overlays uses flooding. The useof a structured P2P overlay as the substrate for RDSpace isdiscussed from the aspect of query efficiency, resourcedecoupling, event handling, and consistency.

2.2.1 Query efficiency

Template matching is required in resource discovery inorder to support flexible and powerful queries. The abstracttuple space interface expects that the scalability of thesystem to have a complexity of O(1). The primary goal ofresource discovery is to guarantee the discovery ofresources. Therefore definite recall in resource discoveryis absolutely required. Structured P2P overlays guaranteethat the discovery of resources with an overhead of O(log N)leading to a good scalability. Unstructured overlays arecompletely different, when their characteristics are com-pared with the requirements of the tuple space interface.They offer powerful query functionality but poor scalabilityand no guarantees in resource discovery.

2.2.2 Resource decoupling

Tuple spaces require that resources and resources descrip-tions are persistent as this allows the decoupling of resourceproviders and requesters spatially and temporally. BothFig. 1 RDSpace architecture

Peer-to-Peer Netw Appl (2009) 2:283–297 285

structured and unstructured P2P overlays support persistentstorage of resources and their descriptions. Structured P2Poverlays allow decoupling of resources and resourcesdescriptions, as resource providers may not host thedescriptions of their resources. On the other hand automaticreplication of resources and their descriptions in structuredP2P overlays is easier than that in unstructured P2Poverlays as the former determines which peers nodes areused for replication.

2.2.3 Event handling

Another requirement of a tuple space interface is eventhandling. Events are important especially in the cases whena client issues an in(pattern) or read(pattern) operation andno tuples are available to match the pattern. Theseoperations have to be blocked until a tuple matching thepattern becomes available. Clients with blocking operationswaiting for tuples have to be informed by an event when anappropriate tuple becomes available. This is not supportedin both unstructured and structured P2P overlays.

2.2.4 Consistency

Consistency mainly deals with the problems of churnsituations. The system has to remain in a consistent statealthough the number of leaving and joining nodes can bevery high in a dynamic network environment. The tuplespace interface requires instant consistency of the system

after nodes have joined or left, which requires that thesystem is actually able to detect and react on both events.

Unstructured P2P overlays directly fulfill the consistencyrequirements in case of leaving and joining nodes.Resources and resource descriptions are stored on the samenode, no replication is used and no information onneighbouring nodes is maintained. However, it is notpossible to detect the events of node joining and leavingin unstructured P2P overlays.

Structured P2P overlays use replications to handle churnsand maintain neighborhood information within an overlaynetwork. For example, Pastry uses leaf sets of neighboringnodes combined with heartbeats to detect joining and leavingnodes. This allows structured P2P overlays to react onchanges in the network topology. On the other hand theseauxiliary structures have to be kept consistent under churns.Thus structured P2P overlays are not instantly consistentafter churns but require a few steps to regain consistency.

2.2.5 Choosing structured P2P overlays for RDSpace

Having considered the above four aspects it becomesobvious that both unstructured and structured P2P overlayshave their individual advantages and drawbacks when theyare used for the implementation of a tuple space forresource discovery. Nevertheless, when all factors are takeninto account, structured P2P overlays are better suited forRDSpace than unstructured P2P overlays. The architectureof a peer node in such a setting is shown in Fig. 2.

Operations Tuple Space Structured P2P Unstructured P2P

Create out (tuple) put (key, data) Not required

Retrieve read (pattern) get (key) Query flooding

Remove in (pattern) remove (key) Local delete

Change out (pattern) put (key, data) Local update

Queries

Exact match Yes Yes Yes

Keywords Yes/No (prototype) No Yes

Wildcard Yes/No (prototype) No Yes

Range queries Yes/No(prototype) No Yes

Definite recall Yes Yes No

Scalability O(1) O(log N) O(N)

Resources

Persistent Yes Yes Yes

Decoupling Yes Yes No

Events Yes with in/red No No

Consistency

Leaving nodes O(1) O (log N) O(1)

Joining nodes O(1) O (log N) O(1)

Detect joins Yes Yes No

Detect leaves Yes Yes No

Table 1 Tuple spaces on struc-tured versus unstructured P2Poverlays

286 Peer-to-Peer Netw Appl (2009) 2:283–297

Each physical node provides 0 to n resources. When anode publishes or retrieves resource descriptions stored onthe global tuple space, the local tuple space interface isaccessed. Hence there exists one virtual tuple spaceinterface for each peer node. The gateway to the physicalnetwork is represented by the DHT that is a structuredrouting overlay. The tuple space adaptation layer is re-sponsible for passing calls against the tuple space interfaceto the underlying DHT implementation. Hence the localtuple space of a node can be represented by the DHTimplementation of the peer node. Figure 3 illustrates how anode publishes and retrieves a resource in RDSpace using astructured P2P overlay.

Let node P represent a resource provider and node Rrepresent a resource requester. Node S represents the nodethat stores the resource description provided by P, hencesevering as a mediator between R and P. When P publishesthe availability of a local resource, it assembles a resourcedescription tuple adhering to the tuple format defined inRDSpace. Then P issues an out(ResourceDescription)operation against its local tuple space interface. The TupleSpace Adaptation Layer processes the tuple and passes it tothe local DHT interface of node P which executes a put(key,value) operation. Depending on the value of the hash key,the resource description is routed to the peer node Sresponsible for that subset of the key space. S receives theresource description through its local DHT interface andstores it on its local peer storage.

Now node R wishes to make use of the resourceprovided by P. R is only aware of the type of the resource.Neither does R know where P is located nor does R knowthat the requested resource is provided by P. Therefore Rassembles a pattern describing the requested resource andissues a read(Pattern) operation against its local tuple spaceinterface. The Tuple Space Adaptation Layer of node R

processes the pattern and forwards it to its local DHTinterface which routes the query to node S. Once node Sreceives the query a match in the local DHT occurs and thecorresponding resource description is loaded from its peerstorage. S sends the resource description back to R. After Rhas received the resource description it retrieves all theinformation required to actually access the resourceprovided by P.

2.2.6 RDSpace on top of Bamboo

Bamboo3 is chosen as the structured P2P overlay forRDSpace. Bamboo employs the same overlay structure asPastry. Bamboo was designed with a specific focus onchurn handling. The main components of Bamboo aredescribed below.

& DHT The DHT provides a standard DHT interface tothe Bamboo routing engine.

& DataManager Data stored to the DHT is maintained bythe DataManager. When storing data on a peer node, theDHT forwards the data to the DataManager which isresponsible for the persistent storage of the values.

& Gateway In order to allow applications that are not partof Bamboo to use the DHT functionality, the Gatewayprovides a remote procedure call (RPC) based interfaceto it.

& Scribe The Scribe represents an implementation of theScribe event notification system [16]. Scribe allowspeer nodes to participate in topic based publishsubscribe systems by using multicast trees for eventnotifications.

Fig. 2 The structure of a tuple space on a P2P overlay

Fig. 3 Resource discovery in a P2P structured tuple space

3 http://bamboo-dht.org

Peer-to-Peer Netw Appl (2009) 2:283–297 287

2.3 RDSpace interface

RDSpace provides the following interface methods.

& String getName() Gets the name associated with thetuple space.

& boolean out(ITuple tuple) Stores a tuple in the tuple space.& ITuple in(ITuple pattern) Retrieves a tuple matching a

pattern from the tuple space. When no matching tuple isavailable, this call is blocked. When multiple tuples areavailable, one is returned non-deterministically.

& ITuple inp(ITuple pattern) Just like in(), except thatthis call is not blocking. When no matching tuple isavailable null is returned.

& ITupleArray ing(ITuple pattern) Just like inp(),except that all matching tuples are returned.

& ITuple in(ITuple pattern) Retrieves a tuple matchingpattern from the tuple space. When no matching tuple isavailable this call is blocked. When multiple tuples areavailable, one is returned non-deterministically.

& ITuple readp(ITuple pattern) Just like read(), exceptthat this call is non-blocking. When no matching tupleis available null is returned.

& ITupleArray readg(ITuple pattern) Just like readp(),except that all matching tuples are returned.

& boolean delete(ITuple tuple) Deletes a tuple from thetuple space.

When compared to the basic tuple space interface ofLinda, the eval() operation is omitted in favor of a delete()operation that can be used to remove published tuples fromthe tuple space. The semantics of the other operations aresimilar. The out() and the delete() operations are non-blocking, they return immediately after they complete theirrequests. Both the read() and the in() operation are blockingcalls that wait until a tuple matching pattern is available inthe tuple space. In addition to these blocking calls, inp() andreadp() provide non-blocking alternatives. While all previ-ously mentioned operations only return a single tuple, ing()and readg() can be used to retrieve multiple matching tuples.

2.4 Churn handling

Churn can be defined as the number of changes that affectthe configurations of a P2P overlay over a particular periodof time. Events that change the configurations are peernodes joining or leaving the network environment, failingpeer nodes, and resources that are added to or removedfrom the network environment.

2.4.1 Churn handling at Bamboo DHT layer

Bamboo is based on the Pastry DHT employing routingtables and leaf sets. Under churns, these data structures

need to regain consistency. Routing tables and leaf setshave to be updated when nodes are joining and leaving. Allissues that deal with change in the physical network meshare handled by the Bamboo DHT. Bamboo requires thatreceivers of messages send acknowledge packets back to thesender. While this increases general network traffic, it has theadvantage that failed nodes are detected quickly. Furthermoreit is possible to detect some unstable nodes by examining themessage latencies. High latencies are an indicator for anetwork congestion or upcoming failure. In both casesmessages are resent by Bamboo to a different neighbor node.

A difficult question in this context is the timeout valuewhile waiting for an acknowledge message. If timeouts areset too long the overall latency increases, if they are tooshort the message overhead increases. Bamboo keeps trackof previous observed latencies and uses them to calculatenew timeout values for that particular connection. In orderto quickly detect replacements for failing nodes, eachBamboo node actively sends a list of its neighbor nodesto its neighbor nodes. This allows nodes to be quicklynotified about newly joined nodes.

2.4.2 Churn handling in RDSpace

Besides the issues associated with changes of the physicalnetwork configuration that are handled by the DHT layer,churns also affect the tuple space layer. For example aresource provided by a peer node P is stored at a node S asdescribed in Section 2.2.5. Under a churn, this setting canlead to two different inconsistent states. When P leaves, Sstill contains the resource information although the resourceprovided by P is no longer available within the networkenvironment. Hence the resource still appears to be valid toother nodes, but it no longer exists. The second inconsistentstate occurs when S leaves and P remains in the networkenvironment. In this case the resource would appear to beno longer available although it is still accessible.

RDSpace associates time-to-live (TTL) with a tuple thatdescribes a resource. When a resource provider publishes atuple for resource description, a TTL value is added to thetuple indicating the availability of the resource in terms ofthe amount of time in seconds. Once TTL has reached avalue of zero, the resource description is purged from thetuple space.

2.5 Storing tuples

When a tuple is stored on the Bamboo routing overlay, ithas to be mapped to the (key, value) representation used byBamboo. Figure 4 outlines the complete sequence ofpublishing a tuple on the Bamboo DHT.

After the tuple has been converted to its DHT (key,value) representation, Bamboo routes it to the peer node

288 Peer-to-Peer Netw Appl (2009) 2:283–297

responsible for the partition of the ID space where the keyis located. The call out() of the tuple space interface isblocked until it receives an answer from the DHT overlaystating if the put() operation is successful. Deleting a tupleis accomplished in a similar way. The call to delete() isblocked until an answer from S has been received.

Tuples are automatically replicated by the underlyingBamboo DHT layer in order to increase the availability oftuples when nodes are failing. Bamboo nodes replicate datato the lightly loaded nodes that are part of their leaf sets.Hence, using a standard leaf set size of 4 in both directions,the replications factor is 8. Replication not only increasesthe availability of tuples, but also lowers the averagelatency and improves load balancing.

2.6 Retrieving tuples

In in() and read() operations against the tuple spaceinterface, a pattern is used to describe the tuple that is tobe retrieved from the resource discovery system. Thesequence of retrieving a tuple matching a pattern from theBamboo DHT overlay is outlined in Fig. 5.

After converting the pattern to its DHT key representation,Bamboo routes the key to the peer node responsible for thatpartition of the ID space. The responsible peer node checks ifthis key is available in its local peer storage and returns any(key, value) pair matching the request key. The calls areblocked until an answer from the DHT overlay is received.

3 RDSpace evaluation

RDSpace has been evaluated from the aspects of scalability,event latency and churn handling. This section presents theevaluation results.

3.1 Experiment setup

During the evaluation of the prototype system, 4 physicalnodes were used and connected by a 100Mbps network.Each instance of Bamboo opens a set of network ports onwhich it is listening for incoming requests of applicationsand other Bamboo peer nodes. This makes it possible toload several instances of Bamboo on a single physicalnode, as long as the network ports used by each instanceare different. Each instance of Bamboo on a physical nodeis called a logical node. Thus a physical node was able tohost between 1 and n logical nodes. For each instance ofBamboo, a separate instance of the Java virtual machine(VM) is launched. Hence the memory footprint required bythe Java VM has to be added to the memory footprintrequired by the resource discovery system to get the totalmemory requirement of a logical node instance. Aninstance of the Java 1.5 SE VM requires 10 MB of mem-ory. Together with the memory requirements of Bambooincluding all stages required for RDSpace of 12 MB, thetotal memory demand for an instance of a logical nodeamounted to 22 MB. During the tests this was a limitingfactor for physical nodes such as Node 3, which had only arelatively small amount of memory.

In order to achieve repeatable and realistic test results,thrashing and swapping of memory pages to the hard diskwere not used. The maximum number of logical nodes thatcould be executed on the most powerful physical node(Node 1) was limited to 64, while the least powerful node(i.e. node 3) was only able to simulate 16 logical nodes.The localhost test setup used only one physical node tosimulate 1 and n logical nodes. All tests using the localhosttest setup were performed on the physical Node 1. Incontrast to the localhost test setup, the LAN setup used allthe 4 physical nodes. The number of logical nodes wasdistributed evenly among the physical nodes. For example,

Fig. 4 RDSpace Out operation sequence

Fig. 5 RDSpace Read operation sequence

Peer-to-Peer Netw Appl (2009) 2:283–297 289

in a simulated network with n logical nodes, each physicalnode hosts n/4 logical nodes.

3.2 Evaluation results

3.2.1 Scalability

This section presents the evaluation results on scalability ofRDSpace. The particular interests were focused on thescalability of the underlying DHT layer and the scalabilityof the complete RDSpace after the tuple space abstractionhas been added.

Bamboo DHT layer In order to test the scalability of theunderlying DHT layer, the following test scenario wasused. A client application published a randomly generatedinteger value on the DHT by issuing a put() call against theBamboo Gateway which forwards the request to the DHTlayer. After the client application received the response thatthe value was stored to the DHT, it immediately issued aget() call to retrieve the previously stored value. Eachrandomly generated integer value was mapped to a (key,value) pair.

The test was conducted using 5 consecutive runs on onephysical node. The number of logical nodes varied from 1to 64. Each run consisted of 2,500 individual publishing/retrieving cycles. Average values over the 5 runs as well asa normalized value showing the average runtime over the 5runs in relation to the number of logical nodes areillustrated in Fig. 6.

The same test scenario using an identical number oflogical nodes as well as identical test data generation wasalso performed on the LAN setup using the 4 physicalnodes. The averaged and normalized results for the LANsetup are illustrated in Fig. 7.

Bamboo DHT layer shows a logarithmic scalability withan increase in the number of logical nodes used. Thelocalhost setup tests show about 2.5 to 3 times betterperformance than that of the LAN setup. The localhostperformance is superior because the communication be-tween logical nodes is performed on the same physicalnode and no data communication is required. However, thescalability in the LAN setup is better. This can be observedfrom the changes of ratios of runtime to number of nodesused in both the setups. An interesting result of these testsis that the performance of the Bamboo DHT can sufferwhen a few logical nodes run on less powerful nodes. Aspeer nodes participating in a structured P2P overlay such asBamboo may have varied processing capabilities, theruntime scalability is not guaranteed to be logarithmicalthough the number of exchanged messages is guaranteedto be logarithmic.

RDSpace interface layer As RDSpace interface layer isimplemented on top of the Bamboo DHT layer, theoverhead of the tuple space interface should be comparedwith that of the underlying Bamboo DHT layer. TheRDSpace interface layer was tested by publishing randomlygenerated tuples to the system using out() calls. Thesetuples were retrieved using a read() calls. An additionalrandomly generated ID was added to the tuple in order todistribute it randomly over the available logical nodes forstoring tuples. Figure 8 and Fig. 9 show the testing resultsusing one physical node and the 4 physical nodesrespectively. Each shown runtime value represents theaverage value of 5 runs, whereas each run consists of2,500 individual out() and read() cycles.

Testing the RDSpace interface using both the localhostand the LAN setup shows that while the additional tuplespace layer does not damage the scalability of the system, it

Fig. 6 Bamboo-DHT Put/Getperformance on one physicalnode

290 Peer-to-Peer Netw Appl (2009) 2:283–297

nevertheless adds a significant processing overhead on topof it. The RDSpace interface shows a logarithmic scal-ability in both test setups. While the LAN setup still showsa better scalability because of real parallelism, it issurprising that the difference between the two setups isquite small.

However the total runtime of the RDSpace tuple spaceinterface is considerably higher than that of the BambooDHT. Compared with Bamboo DHT, the runtime of theRDSpace in the localhost setup increases 2.32 times, whilein the LAN setup RDSpace shows a 1.45 times higherruntime. This is mainly caused by the time required totransmit a tuple. While an integer used in the DHT tests hasa size of 4 bytes, the size of a tuple used is about 850 bytes.This larger payload is not only transmitted between theTuple Space Interface and the Tuple Space Manager, but

also transmitted between peer nodes during each routingstep. In addition, the communication between the TupleSpace Interface and the Tuple Space Manager adds anotheroverhead on top of the underlying Bamboo DHT Gateway.

Java object serialization employed in the communicationbetween Tuple Space Interface and Tuple Space Manageradds a further overhead. The required communication timebetween the Tuple Space Interface and Tuple SpaceManager is the same regardless of the size of the simulatednetwork. In the localhost tests it is about 8–10 ms, while theLAN tests show a time of about 16–20 ms. As thetransmission time can be neglected for the localhost setup,the results indicate that about 8–10 ms of the requiredcommunication time is caused by Java object serialization.This does not seem unreasonable as tuples are relativelycomplex objects and Java object serialization relies on

Fig. 7 Bamboo-DHT Put/Getperformance in a LANenvironment

Fig. 8 RDSpace Out/Readoverhead on one physical node

Peer-to-Peer Netw Appl (2009) 2:283–297 291

runtime checks using Java reflection in order to detect thestructure of an object and store it in a persistent state.

When the standard deviation is compared with the actualruntime results, it becomes evident that the larger thenumber of logical nodes used, the higher the deviation fromthe expected runtime values. This is especially obvious forthe test using 64 logical nodes as these results are alreadyaffected by page swapping issues. In general the standarddeviation is higher in LAN tests because the parallelismemployed in these tests adds an additional level ofuncertainty to the results.

3.2.2 Event latency

The event latency of RDSpace was evaluated by testingboth direct and mediator based notifications. Two applica-tions were used. Application A tried to read a tuplematching a pattern with a randomly generated ID and wasblocked when such tuple was available. Then application Bpublished a tuple matching the pattern that application Awas waiting for. As application A subscribed to be notified ofthe availability of such a tuple, it received an event wheneverapplication B published its tuple. The event latency is atime span between the time that application B publishedthe tuple and the time that application A received it.

Tests of both direct notifications and mediator-basednotifications were conducted using both the localhost andLAN setups. The results are illustrated in Fig. 10. Each testrepresents the average value of 1,000 event notification andpublication cycles. As tests only measure the time betweenpublication and reception, the latency includes a completeput() cycle and a resource announcement performed byapplication B, the event distribution via Scribe and acomplete get() call cycle executed by application A. When

the results using 64 nodes are omitted because of pageswapping reasons, it can be observed that both direct andmediator notifications are nearly identical in the two setups.However the mediator-based solution in the LAN setupperforms slightly better than that of the local setup as theload is distributed among more nodes.

3.2.3 Churn resilience

This section presents the evaluation results of churnhandling of RDSpace. The most important aspect in thissetting is the percentage of successfully retrieved tuples interms of the number of leaving and failing nodes. Churnresilience was tested by publishing 1,000 randomlygenerated tuples on RDSpace using 32 logical nodes.Failing and leaving nodes were simulated by terminatinga certain number of logical nodes. Depending on thenumber of failing nodes, only a certain percentage of themcan be retrieved successfully. This group of tests was onlyperformed using the localhost setup, as the performancewas not concerned in this context. Figure 11 shows thetesting results of the three groups respectively.

The first group depicts the expected theoretical success-ful retrieval rate when no replication is used in RDSpace.As the tuples are distributed uniformly among the nodes,the number of tuples that can be successfully retrieved isdirectly proportional to the number of failing nodes. Eachtuple is only hosted by a single node. If that node fails, thetuple is no longer accessible. The other two groups ofresults show the actual testing results using RDSpace. Twodifferent cases of test scenarios were performed. The firstone simulated instant node failures. In this case a number ofnodes failed simultaneously and instantly. The second testscenario simulated a gradual and sequential node failure.Only a single node failed at a given time.

Fig. 9 RDSpace Out/Readoverhead in a LAN environment

292 Peer-to-Peer Netw Appl (2009) 2:283–297

Using replication increases the resilience of RDSpace interms of churns. The reason is that when nodes carrying theresource information leave the network, the information isstill available and accessible on the replicated nodes.Bamboo uses a replication factor of 8. This means that aresource is replicated on 8 other nodes in addition to thenode responsible for it. These 8 nodes are the neighboringnodes in the node’s leaf set. When nodes fail gradually, theDHT layer is able to detect these failures and start a self-repair procedure. Once the self-repair procedure is finished,the system is back to a consistent state. When a node fails atintervals larger than the time needed for self-repair, thenode is always in a consistent state and all resources storedon it can be retrieved successfully. This is shown in Fig. 11by the testing results for gradual node failure.

However, when nodes fail at intervals smaller than thetime required for self-repair, the RDSpace network may notbe able to reach a consistent state quickly in which allresources can be retrieved successfully. A worse case couldbe a situation where several nodes are failing simultaneous-ly. In this case the whole leaf sets of nodes may failcompletely and instantly. Furthermore, when the number ofnodes that fail simultaneously is equal or larger than thereplication factor, all information on a given resourcecannot be discovered from the overlay network. In case ofsimultaneously failing nodes, the system has no chance tostart a self-repair process as the ID space is not availableinstantly. This is shown in Fig. 11 with the results of instantnode failures. As long as the number of failing nodes issmaller than the replication factor, RDSpace is able to

Fig. 11 Churn resilience ofRDSpace

Fig. 10 RDSpace event latency

Peer-to-Peer Netw Appl (2009) 2:283–297 293

retrieve all tuples. Furthermore, when 16 of 32 nodes failinstantly, RDSpace is still able to retrieve around 90% ofthe stored tuples.

4 Related work

Table 2 compares RDSpace with JavaSpace, IBM T-Space4,Comet [17], Lime [18], Panda [19], PeerSpaces [20], andPeerGameSpace (PGS) [21]. Among them, JavaSpace andIBM T-Space build on a client-server structure, RDSpace,Comet, Panda and PGS use a structured P2P infrastructure,Lime and PeerSpaces use an unstructured P2P overlay.

Comet represents a content-based coordination infra-structure on top of a structured P2P network. Cometfocuses more on the actual coordination requirements ofinteracting processes than on resource discovery. Hence ithas no particular use cases. Just like Comet, Panda does notaim at a particular use case but serves as the basis forresearching the deployment of a traditional Linda systeminto a truly distributed environment. In [19] Panda isdescribed with a theoretical evaluation. Hence it is notcertain if an implementation of the Panda system hasalready been developed as neither a prototype system norpractical evaluation results are available. Similar toRDSpace, PGS also builds a tuple space model on top ofa structured P2P overlay. However, PGS is not designed forresource discovery but for coordination and development ofonline multiplayer games in fully distributed environments.In addition, RDSpace uses Bamboo as the P2P overlay, butPGS is built on Typhoon [22]. One direct benefit ofRDSpace over PGS lies in its capability in dealing withchurn situations which can be reflected by the resultspresented in Section 3.2.3.

In order to increase scalability and to guarantee recall inresource discovery, several approaches employ a tuplespace interface on top of a structured P2P routing overlay.These include Comet, Panda, PGS and RDSpace. Comet isthe only system that uses XML for tuple formatting. Allother systems including RDSpace use Java objects toformat tuples limiting the usage of these systems to Javalanguage.

A global memory abstraction provided by the Lindamodel is unstructured. Only Panda and RDSpace establish astructure in the tuple space by aggregating tuples having thesame format. RDSpace uses a virtual table while Panda putsall tuples having the same signature into a file stored on thePAST distributed file system [23]. Aggregating similartuples can speed up the process of matching tuples.

Blocking calls for in() and read() are supported by allsystems, but they differ in the way they implement these

operations. Of the systems employing structured P2Poverlays, Panda, PGS and RDSpace use a Scribe likereverse path-forwarding scheme to establish a multicasttree. Instead of subscribing to an event group maintained bya Scribe like multicast protocol, Comet sends queries tonodes that potentially store the queried tuple and waits forthe first answer it receives. This may have disadvantagesunder churns.

Support for complex queries is limited in most systems.Nearly all of them support only standard Linda queries inwhich a tuple field is either an actual value or a formalvalue that has to be retrieved by the query. JavaSpace,Lime, PeerSpaces, RDSpace and PGS belong to thiscategory. IBM T-Spaces supports arbitrarily complexqueries by using wildcards, regular expressions, and rangequeries. Complex queries in Comet are limited to wildcardand keyword queries. Comet uses Hilbert Space-FillingCurves (HSFCs) provided by the Squid P2P data discoverysystem described in [24] and [25]. HSFCs enable wildcardand keyword based queries in structured P2P overlays bymapping a multi-dimensional query space onto a one-dimensional ID space. Each query parameter adds anadditional dimension to the query space. A query definesa cluster in the query space. By mapping this cluster onto aset of peer nodes with consecutive IDs, it is possible todetermine those nodes that potentially store tuples matchingthe query. Hence each complex query maps to a distinct setof peer nodes to which the query is sent for evaluation.Panda implements queries by distributing them to the peernode that stores the PAST file responsible for that tuplesignature. The peer node matches the query locally usingthe PAST file and returns the first matched tuple.

Churn handling is a critical issue when building tuplespaces on top of structured P2P overlays. The capability ofRDSpace in churn handling lies in two aspects. On the onehand, RDSpace builds on Bamboo using replications tohandle churns. On the other hand, RDSpace associates TTLvalues with tuples. As shown in Section 3.2.3, replicationsstrengthen churn resilience. Panda also uses replications tohandle churns. However, RDSpace makes replications at atuple level which is distinct from the file level replicationsmade by Panda. It is expected that more replications couldbe made in RDSpace leading to a better performance inchurn handling than Panda.

5 Conclusion and future work

In this paper we have presented RDSpace which can beused as scalable information services for resource discoveryin grid computing environments. RDSpace utilises a tuplespace for coordination and communication of peer nodesand employs a structured P2P overlay for routing messages4 http://www.almaden.ibm.com/cs/TSpaces

294 Peer-to-Peer Netw Appl (2009) 2:283–297

Tab

le2

Acomparisonof

RDSpace

with

relatedwork

Tup

lespace

Java

spaces

T-Spaces

RDSpaces

Com

etLim

ePanda

PeerSpaces

PGS

Architecture

Centralized

Centralized

Partitioned,

Replicate(Tup

le)

Partitioned

Partitioned

Partitioned,

Replicate(Files)

Replicated

Replicated

Tup

leform

atJava

objects

Java

objects

Java

objects

XMl

Java

objects

Java

objects

XML

Java

objects

Structure

Unstructured

Unstructured

Structured(virtual

tables)

Unstructured

Unstructured

Structured(Files)

Unstructured

Unstructured

Out,Read,

In,Delete

Yes

(noDelete)

Yes

Yes

Yes

(nodelete)

Yes

(nodelete)

Yes

Yes

(nodelete)

Yes

(nodelete)

Group

operations

Yes

Yes

Yes

No

Yes

No

No

No

Probe

operations

Yes

Yes

Yes

No

Yes

Yes

No

No

Com

plex

queries

Lim

ited

Yes

Lim

ited(prototype),

Yes

(Con

cept)

Lim

ited

Lim

ited

Yes

(?)

Lim

ited

Lim

ited

Recallgu

arantee

Yes

Yes

Yes

Yes

No

Yes

No

Yes

Transactio

nsYes

Yes

No

No

No

No

No

No

Networklayer

Paradigm

Client/Server

Client/Server

P2P

P2P

P2P

P2P

P2P

P2P

Architecture

Client/Server

Client/Server

Structured

Structured

Unstructured

Structured

Unstructured

Structured

Rou

ting

Client/Server

Client/Server

Bam

boo

Cho

rdIP

Network

Pastry

JXTA

Typ

hoon

Tup

leStorage/Loo

kup

Client/Server

Client/Server

Virtual

tables

HSFC/Squ

idIP

Multicast

Files

Broadcast

Sho

rtcuts

Events,Notifications

Client/Server

Client/Server

Scribe

FCFS

IPMulticast

Scribe

Broadcast

Scribe

Chu

rnHandling

TTL/for

Tup

les

TTLforTup

les

Bam

boochurnhand

ling,

TTLforTup

les,Replication

Non

eNot

requ

ired

(Unstructured)

Replication

Not

requ

ired

(Unstructured)

Non

e

Peer-to-Peer Netw Appl (2009) 2:283–297 295

combining the benefits of the two paradigms. The evalua-tion results showed that RDSpace is scalable in resourcediscovery and is resilient in churn handling.

However, the RDSpace prototype is still in its infancy.For example, RDSpace only supports direct match queries.Hence applications using the prototype need to know theresources they wish to search for. While this does not limitthe general usability of the prototype system in resourcediscovery, this limits its applicability in a wide scope. UsingJava objects to format tuples in RDSpace restricts its usageto Java language. Although RDSpace is capable of handlingchurns, it requires a minimum of O(logN) steps to be in aconsistent state which is not fast enough in reaction whenthe number of nodes in the network is large.

One future work on RDSpace will be the support ofcomplex query formats. Security is not considered in theprototype. To apply RDSpace in real-world applications, asecurity mechanism dealing with secure joining andauthentication as well as secure tuple publishing and storingin open grid computing environments has to be developed,implemented and evaluated.

Acknowledgement The authors would like to thank Jochen Palmerfor his contribution to the work.

References

1. Schopf JM, Pearlman L, Miller N, Kesselman C, Foster I, D’ArcyM, Chervenak A (2006) Monitoring the grid with the GlobusToolkit MDS4. Journal of Physics: Conference Series 46:521–525

2. Cooke AW et al (2004) The relational grid monitoring architec-ture: mediating information about the grid. Journal of GridComputing 2(4):323–339

3. Cai M, Frank M, Chen J, Szekely P (2004) MAAN: a multi-attribute addressable network for grid information services.Journal of Grid Computing 2(1):3–14

4. Groep DL, Templon J, Loomis C (2006) Crunching real data on thegrid: practice and experience with the European DataGrid. Concur-rency and Computation: Practice and Experience 18(9):925–940

5. Zhang X, Freschl JL, Schopf JM (2007) Scalability analysis ofthree monitoring and information systems: MDS2, R-GMA, andHawkeye. J parallel distrib comput 67(8):883–902

6. Milojicic D S, et al (2002) Peer-to-peer computing. TechnicalReport, HPL-2002-57, HP Labs

7. Stoica I et al (2002) Chord: a scalable peer-to-peer lookupprotocol for Internet applications. IEEE trans netw 11(1):17–32

8. Rowstron A, Druschel P (2001) Pastry: scalable, distributed objectlocation and routing for large-scale peer-to-peer systems. In: Proc.of IFIP/ACM Int’l Conference on Distributed Systems Platforms(Middleware), pp. 329–350

9. Ratnasamy S, Francis P, Handley M, Karp R M, Shenker S (2001)A scalable content-addressable network. In: Proc. of SIGCOMM,pp. 161–172

10. Foster I, Iamnitchi A (2003) On death, taxes, and the convergenceof peer-to-peer and grid computing. In: Proc. of the 2ndInternational Workshop on P2P Systems, pp. 118–128

11. Talia D, Trunfio P (2003) Toward a synergy between P2P andgrids. IEEE Internet Computing 7(4):94–96

12. Rhea S, Geels D, Roscoe T, Kubiatowicz J (2004) Handling churnin a DHT. In: Proc. of USENIX Annual Technical Conference, pp127–140

13. Oppenheimer D, Albrecht J, Patterson D, Vahdatm A (2004)Distributed resource discovery on Planetlab with SWORD. In:Proc. of the 1st workshop on Real, Large Distributed Systems,2004

14. Bharambe A R, Agrawal M, Seshan S (2004) Mercury: supportingscalable multi-attribute range queries. In: Proc. ACM SIGCOMMConference on Applications, Technologies, Architectures, andProtocols for Computer Communication, pp. 353–366

15. Gelernter David (1985) Generative communication in LINDA.ACM Comput Surv 7(1):80–112

16. Rowstron Antony, Kermarrec Anne-Marie, Castro Miguel, DruschelPeter (2001) Scribe: the design of a large-scale event notificationsystem. Microsoft Research, Cambridge

17. Li Z, Parashar M (2005) Comet: a scalable coordination space fordecentralized distributed environments. In: Proceedings of 2ndInternational Workshop on Hot Topics in Peer-to-Peer Systems,San Diego, CA, USA

18. Murphy Amy L, Picco Gian Pietro, Roman Gruia-Catalin (2006)LIME: a coordination model and middleware supporting mobilityof hosts and agents. ACM Trans Softw Eng Methodol 15(3):279–328

19. Christian Anne, Duarte Marco, Nielson Seth, Pound Andrea,Sandler Dan (2004) Panda: An implementation of generativecommunication on a structured peer-to-peer overlay. Rice Uni-versity, Technique Report

20. Nadia Busi, Cristian Manfredini, Alberto Montresor, GianluigiZavattaro (2003) PeerSpaces: data-driven coordination in peer-to-peer networks. In: Proceedings of the 18th Annual ACMSymposium on Applied Computing, pp. 380–386

21. Wang Chia-Wei, Hsiao Hung-Chang, Sun Wen-Hung, KingChung-Ta, Sun Ming-Tsung (2007) Building a tuple space onstructured peer-to-peer networks. The Journal of Supercomputing40(3):213–227

22. Hsiao Hung-Chang, King Chung-Ta, Wang Chia-Wei (2005)TYPHOON: mobile distributed hash tables. J parallel distribcomput 65(2):191–206

23. Antony Rowstron and Peter Druschel (2001) Storage managementand caching in PAST, a large-scale, persistent peer-to-peer storageutility. In: Proceedings of the 18th ACM SOSP’01, Lake Louise,Alberta, Canada

24. Cristina Schmidt and Manish Parashar (2004) Analyzing thesearch characteristics of space filling curve-based indexing withinthe squid P2P data discovery system. Rutgers University,Technique Report

25. Schmidt Cristina, Parashar Manish (2004) Enabling flexiblequeries with guarantees in p2p systems. IEEE Internet Computing8(3):19–26

296 Peer-to-Peer Netw Appl (2009) 2:283–297

Dr. Maozhen Li is a Lecturer in the School of Engineering andDesign at Brunel University, UK. He received the PhD in 1997 fromInstitute of Software, Chinese Academy of Sciences, Beijing. Hejoined Brunel University as a full-time lecturer in 2002. His researchinterests are in the areas of grid computing, intelligent systems, P2Pcomputing, semantic web, information retrieval, content based imageretrieval. He has over 60 scientific publications in these areasincluding journals of IEEE Transactions on Knowledge and DataEngineering, IEEE Transactions on Neural Networks, IEEE Distrib-uted Systems. He authored “The Grid: Core Technologies”, a well-recognized textbook on grid computing which was published byWiley in 2005. He has served the Technical Programme Committee ofover 30 conferences. He is on the editorial boards of Encyclopedia ofGrid Computing Technologies (IGI) and Applications and theInternational Journal of Grid and High Performance Computing(IGI). He is the course director for MSc in Distributed Computing

Systems Engineering. He is a member of IEEE. Contact him atElectronic and Computer Engineering, School of Engineering andDesign, Brunel University, Uxbridge, UB8 3PH, United Kingdom;Maozhen.Li@brunel.ac.uk.

Dr. Man Qi is a Lecturer in Dept of Computing at Canterbury ChristChurch University, UK. She was a Research Fellow in Dept. ofComputer Science at University of Bath, UK from Jan. 2001 to Oct.2003. Her research interests are in the areas of Computer Graphics,Computer Animation, Multimedia and Grid Computing Applications.Contact her at Department of Computing, Canterbury Christ ChurchUniversity, Canterbury, Kent, CT1 1QU, United Kingdom; mq4@can-terbury.ac.uk.

Peer-to-Peer Netw Appl (2009) 2:283–297 297