8
J. Parallel Distrib. Comput. 65 (2005) 1462 – 1469 www.elsevier.com/locate/jpdc Multiuser 3D virtual simulation environments support in the Gnutella peer-to-peer network Azzedine Boukerche a , , Regina B. Araujo b , Marcelo Laffranchi b a SITE, University of Ottawa, Canada b DC, Universidade Federal de São Carlos, SP, Brazil Received 5 January 2004; accepted 30 May 2005 Abstract One of the challenges for 3D multiuser virtual simulation environments (3DMUVEs) developers is to keep the shared virtual simulation environment synchronized among all the participating users’ terminals. Support to 3DMUVEs through traditional client–server commu- nication model offers simpler management but can lead to bottlenecks and higher latencies. Peer-to-peer communication model, on the other hand, offers no central coordination but are more complex to manage. Current peer-to-peer networks, such as KaZaA and Gnutella, provide multimedia sharing services but do not support multiuser 3D virtual environment (VE) applications. This paper describes a solution to support 3DMUVEs in a hybrid peer-to-peer Gnutella network, which provides session control and distributed shared VE synchronization. As a result of this work, two components specified by the ongoing multiuser extension to the MPEG-4 standard were implemented and integrated to the Gnutella network for control and synchronization. This solution minimizes the disadvantages of client–server and pure peer-to-peer models. The results show that this approach can be a feasible solution, specially for spontaneous 3DMUVEs that can emerge from any user, with no investment needed (apart from his own computer). The use of peer-to- peer networks such as the Gnutella could be used as a test environment for companies wishing to check both their multiuser 3DMUVEs software for correctness and their acceptance by the users community before making heavy investments. © 2005 Published by Elsevier Inc. Keywords: Collaborative virtual environment; Distributed virtual simulation; Peer-to-peer networks; Synchronization; Gnutella network 1. Introduction The synchronization of 3D virtual simulation environ- ments that are shared among multiple users (3D multiuser virtual environments—MUVEs) in a network is a challenge for developers. Any change made to the environment by one participating user has to be broadcasted to all other Dr. A. Boukerche’s work was partially supported by the Canada Re- search Chair Program, NSERC, the Canadian Foundation for Innovations Funds and Ontario Innovation Trust Funds (OIT/Distinguished Researcher Award). Dr. R.B. Araujo’s work has been partially sponsored by CAPES, a Brazilian sponsorship agency. Corresponding author. E-mail addresses: [email protected] (A. Boukerche), [email protected] (R.B. Araujo), [email protected] (M. Laffranchi). 0743-7315/$ - see front matter © 2005 Published by Elsevier Inc. doi:10.1016/j.jpdc.2005.05.014 participant users who share the same virtual environment (VE). For that, efficient communication models, such as the traditional client–server model, are used to support the exchange of information among users’ client terminals. The advantages of the client–server model are well known and include simpler management and billing model as well as simpler protection against frauds. However, the model cen- tered on a server presents single point of failures and can be subjected to bottlenecks. The peer-to-peer communica- tion model can overcome these pitfalls by providing direct collaboration among all nodes in a network, without a cen- tral coordination and no single point of failure. However, this is a distributed solution and therefore, more complex to manage [14,20]. Examples of 3DMUVEs implemented as peer-to-peer applications include NPSNET [12], MI- MAZE [3], MASSIVE [6] and DIVE [7]. Hybrid solutions

Multiuser 3D virtual simulation environments support in the Gnutella peer-to-peer network

Embed Size (px)

Citation preview

J. Parallel Distrib. Comput. 65 (2005) 1462–1469www.elsevier.com/locate/jpdc

Multiuser 3D virtual simulation environments support in the Gnutellapeer-to-peer network�

Azzedine Boukerchea,∗, Regina B. Araujob, Marcelo Laffranchib

aSITE, University of Ottawa, CanadabDC, Universidade Federal de São Carlos, SP, Brazil

Received 5 January 2004; accepted 30 May 2005

Abstract

One of the challenges for 3D multiuser virtual simulation environments (3DMUVEs) developers is to keep the shared virtual simulationenvironment synchronized among all the participating users’ terminals. Support to 3DMUVEs through traditional client–server commu-nication model offers simpler management but can lead to bottlenecks and higher latencies. Peer-to-peer communication model, on theother hand, offers no central coordination but are more complex to manage. Current peer-to-peer networks, such as KaZaA and Gnutella,provide multimedia sharing services but do not support multiuser 3D virtual environment (VE) applications.

This paper describes a solution to support 3DMUVEs in a hybrid peer-to-peer Gnutella network, which provides session control anddistributed shared VE synchronization. As a result of this work, two components specified by the ongoing multiuser extension to theMPEG-4 standard were implemented and integrated to the Gnutella network for control and synchronization. This solution minimizes thedisadvantages of client–server and pure peer-to-peer models. The results show that this approach can be a feasible solution, specially forspontaneous 3DMUVEs that can emerge from any user, with no investment needed (apart from his own computer). The use of peer-to-peer networks such as the Gnutella could be used as a test environment for companies wishing to check both their multiuser 3DMUVEssoftware for correctness and their acceptance by the users community before making heavy investments.© 2005 Published by Elsevier Inc.

Keywords:Collaborative virtual environment; Distributed virtual simulation; Peer-to-peer networks; Synchronization; Gnutella network

1. Introduction

The synchronization of 3D virtual simulation environ-ments that are shared among multiple users (3D multiuservirtual environments—MUVEs) in a network is a challengefor developers. Any change made to the environment byone participating user has to be broadcasted to all other

� Dr. A. Boukerche’s work was partially supported by the Canada Re-search Chair Program, NSERC, the Canadian Foundation for InnovationsFunds and Ontario Innovation Trust Funds (OIT/Distinguished ResearcherAward). Dr. R.B. Araujo’s work has been partially sponsored by CAPES,a Brazilian sponsorship agency.

∗ Corresponding author.E-mail addresses:[email protected](A. Boukerche),

[email protected](R.B. Araujo), [email protected](M. Laffranchi).

0743-7315/$ - see front matter © 2005 Published by Elsevier Inc.doi:10.1016/j.jpdc.2005.05.014

participant users who share the same virtual environment(VE). For that, efficient communication models, such asthe traditional client–server model, are used to support theexchange of information among users’ client terminals. Theadvantages of the client–server model are well known andinclude simpler management and billing model as well assimpler protection against frauds. However, the model cen-tered on a server presents single point of failures and canbe subjected to bottlenecks. The peer-to-peer communica-tion model can overcome these pitfalls by providing directcollaboration among all nodes in a network, without a cen-tral coordination and no single point of failure. However,this is a distributed solution and therefore, more complexto manage[14,20]. Examples of 3DMUVEs implementedas peer-to-peer applications include NPSNET [12], MI-MAZE [3], MASSIVE [6] and DIVE [7]. Hybrid solutions

A. Boukerche et al. / J. Parallel Distrib. Comput. 65 (2005) 1462–1469 1463

combine the advantages of both centralized and distributedapproaches. Mauve and colleagues[13], describe a hybridsolution based on proxies which are used as an extension tocentral servers. The proxies have some of the servers func-tionalities and are located close to the users’ nodes. Althoughproxies approach can improve performance of web services,it remains to be seen if it can be as effective for highly inter-active 3DMUVEs, such as multiplayer games for instance,which demand real-time synchronization among all partici-pants (proxies can add some latency). This paper describesthe implementation and evaluation of a solution to support3DMUVEs in a hybrid peer-to-peer network in which newapplications are spontaneously made available by the differ-ent nodes of the network. The target network for this workis the Gnutella network [4], which is accessed through theLimewire client software [10]. Gnutella network has a de-centralized nature, what makes it more resistant to softwarefailures and malicious attacks, besides having a simple andbasic protocol, that can be used in different experiments,and a very active development community. The Gnutellaversion used in this work contains special nodes, called Ul-trapeers, with larger processing power and network band-width. As a result of this work, the Limewire client soft-ware was extended to support games search, as an exampleof 3DMUVEs. The Gnutella network was also extended tosupport session control and shared VE synchronization. Thecomponents responsible for session control and synchro-nization are compliant to the ongoing multiuser extensionto the MPEG-4 standard to support 3DMUVEs. MPEG-4 isan ISO/IEC Standard for coding and delivery of differentmedia formats in a wide variety of networks and computingplatforms. The results show the feasibility of 3DMUVEs inhybrid peer-to-peer networks.

The paper is organized as follows: related work on peer-to-peer networks and 3D VE is summarized in Section 2.Section 3 briefly describes the Gnutella network. Multiuserextension to the MPEG-4 standard is described in Section4. A hybrid peer-to-peer solution to support 3DMUVEs isintroduced in Section 5—this solution is illustrated with amultiplayer game scenario. Section 6 presents the imple-mentation and evaluation of the system, followed by con-clusions and references.

2. 3DMUVEs requirements

3DMUVEs have appealing applications, from militarytraining to entertainment. Multiplayer games are good exam-ples of challenging 3DMUVEs, where consistency, respon-siveness, scalability, security, reliability and persistency areimportant issues. Current reported solutions typically con-template some of the issues but not all of them. Consistency,responsiveness, security and persistency can be more easilyachieved through client/server solutions and server clusterssolutions, where control and synchronization is centralized.However, as discussed in the introduction, client–server has

scalability and reliability limitations (when there are notreplicated servers).

On the other hand, because of the inherent distributed na-ture of peer-to-peer solutions, these can achieve scalabilityand reliability more naturally. Other issues, such as respon-siveness, can be achieved when they are specially addressed.Security and persistency, however, are difficult requirementsto meet in a peer-to-peer solution.

Many peer-to-peer solutions exploit the area of interest(AOI) to address scalability[21,8,2] by which a user ex-change information only with users that are within his AOI.In this approach, the virtual world can be divided into spa-tial areas (regions, rooms, etc), where each area can be as-signed a multicast group. Users in that area automaticallyjoins the corresponding multicast group. An analysis of thelimitations of region based peer-to-peer approaches can befound in [1,21]. Basically, the more regions there are, themore multicast groups are used (if a user moves from onearea to another, a join/leave operation has to be issued—when multicast is implemented at the network level, thiscan be an expensive operation since latency for joining canbe high. Moreover, depending on the size of the area, un-necessary and redundant data can be sent (the smaller theregion, the larger the number of regions necessary to coverthe whole virtual world, what means more multicast groups.On the other hand, if regions are large, unnecessary infor-mation might be sent to users within that area that receiveinformation from users they are not interested in.

Another important issue to peer-to-peer networks is to ef-ficiently locate the node with the data of interest. One solu-tion is to use distribute hash tables (DHT), by which givena key, a node is located. Peer-to-peer applications can bebuilt on top of overlay networks [9]. A problem with DHTsolutions is that they build an overlay network in which apath between any two nodes can present much higher laten-cies than a path on the underlying network [25]. DHT-basedsolutions include CAN [16], Chord [23], Pastry [19], andTapestry [26]. In a 3DMUVE, the environment is sharedamong multiple users. For that, replicas of the VE can bedistributed to all user’s hosts either fully or partly. As a userenters a VE replica, he can choose his avatar representa-tion and the position he wants to be on the world (alterna-tively, a default position can be assigned). Replicas of theshared environment as well as of the avatars are held onthe remote participating users’ hosts. When an action is per-formed by a user, this action has to be sent to all replicas ofthe environment, so that all users have the same view of theshared VE throughout the application. Consistency mainte-nance requirements may vary depending on the application:collaborative tasks, such as cooperative modeling of sharedobjects, requires strict consistency maintenance against dy-namic real-time interaction applications, such as games, thataccepts a more relaxing consistency maintenance but re-quires an immediate update of the replicas. In this paper,the focus is on the latter type of applications, such as edu-cational, training and entertainment games.

1464 A. Boukerche et al. / J. Parallel Distrib. Comput. 65 (2005) 1462–1469

This paper presents a hybrid peer-to-peer solution inwhich spontaneous 3DMUVEs may emerge and be ex-changed, just like files are exchanged in peer-to-peernetworks such as KazaA. An interesting point about thissolution is that it can offer companies the possibility ofpre-launching 3D VEs as test, where free versions of thesoftware can be made available to be tested by the peer-to-peer community against flaws. The acceptance rate couldbe measured too before eventual substantial investments aremade to the product. Next section introduces the Gnutellanetwork.

3. The hybrid peer-to-peer Gnutella network

When the Gnutella network emerged, back in the year2000, it was considered as the Napster next generation, thenetwork that started the open sharing “fever” of music filesand led the main music record companies to an alert stateabout copyright issues.

As most peer-to-peer file-sharing applications, Gnutellashould, ideally, meet the following requirements[17]: scal-ability (constant response time, high search throughput andincreasing file availability in the presence of a growingnumber of participants); dynamic operability (transparentoperation against dynamic joining and leaving of hosts tothe network); anonymity (identity preservation of partic-ipants seeking or providing information); and high relia-bility (application is not impaired if one or more nodesare disabled).

However, it was soon discovered that Gnutella generateda lot too much network traffic [18]. It was then reorga-nized, mirrowing more efficient structures, such as the Fast-Track (proprietary protocol used by the KazaA application),and incorporating concepts such as the Ultrapeers [22]. TheUltrapeers are special nodes with larger processing powerand network bandwidth capacity. These nodes relieve othernodes of the network from receiving most of the messagestraffic and so reducing the total traffic of the network, mak-ing it faster. With the incorporation of the ultrapeers conceptto the Gnutella network, it started to be considered a hybridpeer-to-peer network.

In order to use the Gnutella network, a user node needs anapplication to provide the connection to other nodes, form-ing an ad hoc network, with no central control, which actseither as a client or as a content provider. One of the bestknown access interface software to the Gnutella network isLimewire [10]. Since there are no centralized servers, whena user wants to enter in the Gnutella network, he initiallyconnects to one of the several existing Ultrapeers, almostalways available. These machines then forward informationsuch as IP addresses and network port to other Gnutellanodes (also known as “peers”). Once connected to the net-work, the nodes interact among themselves by exchangingmessages of the following types [5]:Ping,Pong,Query,RichQuery, Query ResponseandGet/Push.

Characteristics of the Gnutella network, such as the TTL(Time to Live) counter, prevents the re-forwarding of mes-sages to other nodes of the network indefinitely. Althoughthe Gnutella network has been slightly overshadowed byother efficient network protocols, such as the FastTrack, itwas chosen as the target network for this work because ofits conceptual simplicity and also because it is an open soft-ware, allowing the realization and evaluation of further ex-periments on the support to 3DMUVEs.

The existence of 3DMUVEs applications support in theGnutella network is unknown to the authors of this paper—most of the shared content in the Gnutella network is re-lated to music files. The implementation of shared MUVEsrequires session control and synchronization—next sectiondescribes the ongoing extension to multiuser in the MPEG-4standard which specifies the support for such requirements.

4. The ongoing extension to multiuser support in theMPEG-4 standard

MPEG-4 is an ISO/IEC (ISO/IEC JTC1/SC29/WG11)standard, developed by the MPEG(Moving Picture ExpertsGroup) for coding and delivery of different media formatsin a wide variety of networks and computational platforms[15]. There are currently some versions of the MPEG-4Player (MPEG-4 client terminal) available for interactiveTV set-top-boxes. The deployment of the MPEG-4 stan-dard in cellular telephones and mobile devices in generalis already on course and it is just a matter of time beforewe can see complex multimedia applications in these de-vices, through the MPEG-4 Player. The emerging extensionto the MPEG-4 which supports 3DMUVEs, MPEG-4MU,is widely based on thePilot/Dronemechanism proposed bytheLivingWorldsspecification [11]. According to that spec-ification,Pilotsare master copies of shared objects in a VE.Any changes in thePilot state or behavior are replicatedto other instances of that object which are namedDrones(Pilots replicas). The replication of changes in thePilot, orany other object in the graphics scene, can be realized ba-sically through the BIFS-Command Protocol. When thesecommands arrive at the remote users’ client terminals, theyare locally processed, causing the update of the scene, sothat all users that receive a BIFS command have a consistentview of the shared VE.

The MPEG-4MU architecture components responsiblefor managing multiuser sessions and scene synchroniza-tion are the MUTech session controller (MSC) and theMUTech bookeeper (MBK) components, respectively [24].The MPEG-4MU defines a standard interface for the accessto these components and suggest the set of functions thesecomponents should perform—it is then left to developershow these functions are implemented. Next section showshow the MSC and MBK components can be used to supportsession control and synchronization in 3DMUVEs.

A. Boukerche et al. / J. Parallel Distrib. Comput. 65 (2005) 1462–1469 1465

Ultrapeer

Ultrapeer

UltrapeerMSC

Internet

3G Data network

UltrapeerMBK

A

B

C

E

PDA

Ultrapeer

Ultrapeer

UltrapeerMSC

Internet

3G Data network

UltrapeerMBK

A

B

C

E

PDA

Fig. 1. Session control and synchronization components integrated to theGnutella Network.

5. A hybrid peer-to-peer solution to support 3DMUVEs

In the proposed solution to support 3DMUVEs, the MSCand MBK components, compliant to the ongoing extensionto the MPEG-4 standard, are implemented and integratedto the Gnutella network so that a wide range of devicescan access this network through a modified version of theLimewire access interface. Fig.1 illustrates a possible sce-nario. When a user selects a VE, the application and theMBK component are downloaded to his machine. When thedevice has capabilities restrictions, the MBK can be locatedat an Ultrapeer node. The Ultrapeers of the Gnutella networktake over the role of session controllers (MSC component).In the current state of the work, only Ultrapeers nodes canassume this role because they have larger processing powerand network band capacity.

The main tasks of a session controller are, basically, tokeep information about session, zones and users up to date,besides receiving requests for connection from the networknodes and checking if these can apply to become sessioncontrollers Ultrapeers. It would be hard to contemplate theuse of Ultrapeers as session controllers if this task demandedtoo much processing time (after all, Ultrapeer nodes spon-taneously donate their processing power and network con-nection to the realization of the session controlling tasks).

In order to avoid the interruption of a user’s activity inthe shared VE when a session controller Ultrapeer leavesthe network (through a normal or abnormal disconnection),a control transfer scheme was devised which works as fol-lows: if an Ultrapeer leaves the network for any reason, asuccessor Ultrapeer takes over the session so that the ses-sion does not end while there are participating users. Thus,a list of potential session controllers Ultrapeers, ready totake over a session control if the actual controller Ultrapeerleaves the scene, is generated. The list follows the connec-tion order—the next to establish the connection and fulfil therequirements to be an Ultrapeer is the next to take over con-trol. If by any chance the control transfer cannot be realized(for absolute lack of volunteers machines that can be sessioncontrollers), the session then will be terminated (currently,there is always an existing Ultrapeer up and running in theGnutella network for the music file sharing application).

The Ultrapeers selection scheme as successors sessioncontrollers works as follows: when the current Ultrapeer ses-sion controller receives a connection request from a node,it checks first if this node has volunteered as a sessioncontroller and as an Ultrapeer. These options are offeredto the user through the LimeWire client. If the node haslarger processing power and network bandwidth, it is as-signed as session controller number two in the session con-trollers hierarchy, otherwise it enters the session as an or-dinary node. Thus, this node starts receiving session updateinformation and starts managing the current session con-troller status (through heartbeat message). At the moment,up to four successors session controllers are maintained inlatent state (ready to take over the control of a session).If any of these successors session controllers get discon-nected, a new successor is searched for and configured assuch automatically. As soon as the first successor sessioncontroller in the list realizes that the current session con-troller got disconnected (heartbeat message is not receivedfollowed by no reply for a ping), it starts establishing con-nections to all users’ client terminal nodes in that session andbroadcasting them its URL, i.e., the address of the new ses-sion controller. The solution described above can be used tosupport any 3DMUVEs which require session control. Nextsection shows how this mechanism can be used as a sup-port for session control and synchronization in a multiusergame scenario.

5.1. A multiplayer game scenario

Multiuser games support was implemented to illustratehow the solution described above works. When a user wantsto play multiuser games in the Gnutella network, he needs tohave, in his machine, the modified Limewire client softwareto support shared games, besides the 3D Player MPEG-4 andthe game application itself. If this machine is set up as anUltrapeer node, the game application to be downloaded bythe user has, embedded in it, the components responsible forthe session control, and the game synchronization, besidesinformation on existing sessions and players (current numberof players, Ping response time, and if the node is a sessioncontroller Ultrapeer or not).

Once the game is downloaded, an MPEG-4 Player isopened in that user’s client terminal. After downloading andselecting the desired game, another LimeWire window isopened with information on the session and zones for thatgame, besides the number of players. The user can then se-lect the session he wishes to participate. Fig.2 shows theextension made to the Limewire client software to supportthe sharing of games.

The following situations may occur:

• The session controller Ultrapeer leaves the network beforea connection can be established—in this case a messageis shown in the Limewire window and the user will haveto select the session again;

1466 A. Boukerche et al. / J. Parallel Distrib. Comput. 65 (2005) 1462–1469

Fig. 2. The search for games—feature added to the LimeWire code.

• There are no opened sessions for the selected game—theuser may create a session and starts controlling it—soonfriends might join the session;

• The selected session is full—by mirrowing what happensin the chats on the web, a new function will have to beimplemented in the Limewire, which drives the user toanother session automatically—if the user so wishes.

When a session is selected successfully, the user is readyto play. His presence is broadcast (through the session con-troller) to all other players of that session. Any modificationto the game state (caused by users’ interaction), is broad-cast, by the MBK component, to all players, so that syn-chronization is achieved among all participating users. Theperformance of the control transfer mechanism is discussedin the next section, as well as the scalability of the system.

6. Implementation and evaluation of thesynchronization and control transfer mechanisms

The MSC and MBK components were implemented withaccess interfaces compliant to the emerging multiuser exten-sion to the MPEG-4 standard. The MSC component storesthe following data: a list of all zones in a session, and allthe clients of a session (which clients are in what zone)as well as the moment each client subscribed to a zone.The session control is implemented through control mes-sages (MUControl Message) which are divided in controlrequest messages(MUControRequest Messages), control re-ply messages(MUControl Acknowledge Messages)and con-trol command messages(MUControlComman dMessages).These messages were implemented and used for data ex-changing among a session controller Ultrapeer (MSC) and

all the participating users’ client terminals of a VE sessioncontrolled by that Ultrapeer.

For the synchronization of the system, command mes-sages(MUCommand Message)are responsible for request-ing and updating the state of the entities in the shared VE.These messages are divided into request messages(MURe-quest Messages)and update messages(MUUpdate Mes-sages). A request message is a command sent from theclient terminal to theMBK requesting an entity state update.The updating messages were implemented and used by eachMBK of the system as commands sent by the MBK to allremote terminals whose users share the same VE. A ser-vice application was created to generate the users’ updateswhich triggered the creation and transmission of theMUUp-datemessages. Finally, for the session control transfer, newmessages had to be devised since the emerging multiuserextension to the MPEG-4 does not specify an interface forthat kind of service. All messages above are encapsulated,in binary format, before being sent. In order to take part ofa VE, users connect to the Gnutella network, and then lookfor a particular VE to join to a corresponding session andzone. From then on, a test application takes over the taskof randomly generating updates which are transmitted to allparticipating users in the same session and zone, resultingin the synchronization of shared objects. These updates aregenerated in two situations:

1. A client terminal is responsible for“piloting” the sharedobject which is being updated at that terminal—in thiscase,MUUpdatemessages are randomly generated by theclient terminal and processed by the local MBK whichthen sends them to all the replicas (drones) at the remoteclient terminals that share the same VE session, as shownin Fig. 3a where the dotted lines represent theMUUpdatemessages issued. In this case, processing and networktime is saved sinceMURequestmessages (created fromthe interaction of the users) are mapped straight onto theMUUpdatemessages at the client terminal (this is notthe case in a client–server implementation model of theMPEG4MU).

2. A remote client terminal is responsible for“piloting”the shared object which is being updated at anotherterminal—in this case,MURequestmessages are gener-ated randomly and sent from a client terminal to the re-mote client terminal that detains the pilotship, as shownin Fig. 3b as a black line from client terminal 1 to clientterminaln. As a result, the MBK at that terminal gener-atesMUUpdatemessages and send them to all the dronesof the shared object at the remote client terminals, in-cluding the one that provoked the change to the sharedobject, as shown in Fig. 3b as dotted lines.

When the client terminal is a device with very low ca-pability, its MBK can be located at the same Ultrapeer re-sponsible for the session control (MSC). However, this situ-ation is not being evaluated in this paper. The scalability ofthe MBK and the MSC control transfer time were evaluated

A. Boukerche et al. / J. Parallel Distrib. Comput. 65 (2005) 1462–1469 1467

Client Terminal 1

MBK

Client Terminal n

MBK

MBK

user interaction

(pilot)

MBK

Client Terminal n

MB K

MBK

Client Terminal 2

user interaction

(drone)

Client Terminal 2

Client Terminal 1

(a)

(b)

Fig. 3. Shared object synchronization. (a) pilot in local terminal; (b) pilotin remote terminal.

through a series of tests ran in three AMD Duron 950 MHzmachines, with 512 MB of RAM and XP MS/W operatingsystem. The whole system was implemented in Java.

6.1. Scalability of the shared object synchronizationmechanism—MBK

Considering Fig.3a, when the pilotship of a shared objectis at terminal 1, the average delay for the MBK(CTerm-to-CTermP_T), from the moment a user interaction generates anupdating message(procUl_T),sends it to the remote MBKs(net_T),to the moment the remote MBKs process the updat-ing of its shared objects drones(proc-Ur_T), is calculatedas follows:

CTerm-to-CTermP_T= procUl_T+ net_T + proc-Ur_T.

138

230

338

431

535

641

208

298

401

500

598

706

0

100

200

300

400

500

600

700

800

10 20 30 40 50 60

MB

K A

vera

ge

del

ay (

in m

s)

Pilot in Local Terminal Pilot in Remote Terminal

Number of users per session

Fig. 4. Average delay for shared object synchronization.

When the pilotship is at the remote terminal, as shownin Fig. 3b, the client to client updating time(CTerm-to-CTermD_T) is calculated as follows: a user interaction(through a drone) generates an MURequest message (pro-cRl_T), which is sent(net_T) to the remote MBK at theclient terminal that is the pilot of the object being updated.The MBK at this terminal prepares(procUl_T) and sends(net_T)updating messages to all client terminals that sharethe same session (including terminal hosting the drone ofthe referred shared object).

The updating messages are processed in all terminals thatshare that object in the same session(proc-Ur_T).This clientterminal to client terminal updating time is described asfollows:

CTerm-to-CTermD_T= procRl_T+ procUl_T+2 ∗ net_T+ proc-Ur_T

When the number of users in the session increases, sodoes the processing time at each client terminal takingpart of the VE session. Thus, it is important to evalu-ate the scalability of the MBK as it processes and sendsupdating messages and also when it receives updatingmessages for the drones of the shared session. Fig. 4depicts the average delay for the shared objects synchro-nization in the two situations described above, when thenumber of users increases. The network delay averagewas considered as 50 ms (typical average value whenusers are interconnected through the São Paulo Stateacademic network).

The graphic shows that the MBK can scale well interms of the number of users that share one same ses-sion. Sessions of more than 60 users can have accept-able delays when the VE application is not highly in-teractive, as in collaborative editing tasks for instance.However, when the application demands high interactionsuch as in action games, sessions of more than 30 usersmight be unacceptable. These delays may of course de-crease when high speed networks are used (a few tensof milliseconds).

1468 A. Boukerche et al. / J. Parallel Distrib. Comput. 65 (2005) 1462–1469

0

50

100

150

200

250

300

350

400

10 20 30 40 50 60

Number of users per session

MS

C a

vera

ge

con

tro

l tr

ansf

erti

me

(in

ms)

Normal Exit Abnormal Exit

Fig. 5. Session control transfer time among Ultrapeer nodes.

6.2. The performance of the control transfer mechanism

The current MSC controller must be synchronized withthe latent MSCs so that these can be ready to take over con-trol in case the current MSC exits, either in a normal orabnormal way. In a normal exit, the current MSC notifiesthe next latent MSC in the hierarchy that it is leaving, waitsfor acknowledgement and exit. The new MSC controller es-tablishes connection to all other latent MSCs and with allclients of the session it will be controlling. In an abnormalexit, the latent MSCs detect the current MSC controller hasnot sent the heartbeat message (wait for two heartbeat mes-sages time) and send two pings. If no acknowledgement isreceived, the highest latent MSC in the hierarchy assumesthe current MSC is dead and starts taking over control. Eqs.(1) and (2) below represent, respectively, the average delayfor the control transfer in a normal and abnormal exit of thecurrent MSC.

ctrlTT= notif_T+ cnx_TMBK+ cnx_TMSC, (1)

ctrlTT=HB_T+ ping_T+ cnx_TMBK+cnx_TMSC, (2)

where:

• notif_T - time to send an exit notification message andreceive acknowledgement;

• HB_T - heartbeat time-out (the time corresponding to twoheartbeat messages issued from the current MSC con-troller to all latents MSC);

• ping_T - time to issue two pings;• cnx_TMSC- time to establish connection with other latent

MSC in the hierarchy (typically five);• cnx_TMBK - time to establish connections to all client

terminals in a session and send them the new MSC con-troller URL.

The average control transfer time from one session con-troller Ultrapeer to another, is shown in Fig.5.

It can be observed that as the number of users increase,so does the session control transfer time, but quite smoothly.

The delay increase is caused mainly by the connection es-tablishment to the client terminals. The time for the MSCprocessing tasks when the number of clients increase hasa minor influence on the control transfer time since onlya few tasks are performed to complete the control transfer.It can be concluded that the users actively sharing the VEmay not even notice the occurrence of the control transfer,unless some of them decide to leave the session at the timethe transfer is undergoing, or new incoming users are tryingto join a session. Even in these situations, the delay (below350 ms for the worst case of abnormal exit) is quite accept-able and might be unperceptible.

7. Conclusions

It has been registered in the literature that the averageactive uptime for Ultrapeers in the Gnutella network is be-tween 30 and 40 min[22]. However, these numbers reflectonly file sharing applications, which opposed to 3DMUVEs,demand much less user’s interaction time. MUVEs supportin the Gnutella network is something new, so that an evalu-ation of Ultrapeers uptime is aimed as a future work. Also,an informal research is being carried out with the Gnutellausers community, in order to know if the users are sympa-thetic to the idea of allowing their machines to act as sessioncontrollers Ultrapeers (in the same way many users do withUltrapeers for the sharing of multimedia files).

Up to now, this work has concentrated on the session con-trol support and synchronization. The next step is to set upa few VEs and to evaluate the effectiveness of the sessioncontrollers Ultrapeers for more collaborative tasks. This pa-per describes the implementation of a solution to support3DMUVEs in a hybrid peer-to-peer Gnutella network. Ses-sion control and synchronization were implemented as spec-ified by the ongoing multiuser extension to the MPEG-4Standard, adapted to a peer-to-peer model, through the MSCand MBK components. These components were integratedto the Gnutella network, as well as the session control trans-fer mechanism described above. The tests show that the de-lay for session sizes of around 60 users are quite acceptablefor applications that do not demand high rate of interaction,as in collaborative editing tasks for instance. Highly interac-tive 3DMUVEs applications, such as action games, can haveacceptable delays (below 350 ms) for sessions of up to 30users. Important issues such as security and persistency arenot discussed in this paper as it is part of an ongoing work(which makes use of the Ultrapeers node for authenticationand persistency storage information).

The main advantages of this solution over more traditionalsolutions based either on client–server model, pure peer-to-peer or hybrid solutions using proxies are:

• any user who has a 3DMUVE application (a game, forinstance) and wishes to start and control a session maydo so without having to register to a content provider;

A. Boukerche et al. / J. Parallel Distrib. Comput. 65 (2005) 1462–1469 1469

• because the implemented solution uses the MPEG-4Player and the ongoing multiuser MPEG-4 extension,it is easier to make this solution available to differentdevices, from cellular phones to PDAs and set-top-boxes(MPEG-4 is aimed at small mobile devices and networknarrowband, such as the wide area wireless networks);

• the use of Ultrapeers as session controllers providesshorter network latencies for the nodes that connect tothem, as usually the Ultrapeers are located closer to thosenodes;

• a 3DMUVE can be “tested” by the network commu-nity what can be an interesting experiment for companieswishing to check the acceptance rate of a new applicationor game before making heavy investments.

This work has provided a better understanding of the is-sues involved in the implementation of 3DMUVEs in hybridpeer-to-peer networks, and has shown that multiuser supportspecified by the ongoing work of OLGA (OnLine Game)and SNHC (Synthetic Natural Hybrid Coding) groups ofthe MPEG-4 standard can offer a promising solution tothe implementation of multiuser 3D games and MUVEsin general.

References

[1] A. Boukerche, N. J. McGraw, C. Dzermajko K. Lu, Grid-Filtered region-based Data Distribution Management in Large-Scale Distributed Simulation Systems the 38th IEEE/SCS AnnualSimulation Symposium, 2005, pp. 259–266.

[2] C. Bouwens, The DIS vision: a map to the future of distributedsimulation, Institution for Simulation and Training, 1993.

[3] C. Diot, L. Gautier, A distributed architecture for multiplayerinteractive applications on the internet, IEEE Networks Mag. 13 (4)(1999).

[4] Gnutella, 2003,http://www.gnutella.com[5] Gnutella Protocol Specification, version 0.4; available at

http://www.clip2.com/GnutellaProtocol.04. pdf[6] C. Greenhalgh, S. Benford, MASSIVE: a distributed VR system

incorporating spatial trading, IEEE Proceedings of the FifthInternational Conference on Distributed Computing Systems, 1995.

[7] O. Hagsand, Interactive MUVEs in the DIVE system, IEEE Comput.3 (1) (1996).

[8] J. Keller, G. Simon, Solipsis: a massively multi-participant virtualworld, in: Proceedings of PDPTA, 2003, 262–268.

[9] B. Knutsson, H. Lu, W. Xu, B. Hopkins, Peer-to-peer support formassively multiplayer games, in: INFOCOM, March 2004.

[10] Limewire, 2003,http://www.limewire.org[11] LivingWorlds, 1997, http://www.vrml.org/WorkingGroups/living

worlds/draft_2/index.htm[12] M.R. Macedonia, M.J. Zyda, et al., Exploiting reality with multicast

group: a network architecture for large-scale virtual environments,IEEE Comput. Graphics Appl. (September 1995) 38–45.

[13] M. Mauve, S. Fischer, J. Widmer, A generic proxy system fornetworked computer games, in: ACM NetGames, Germany, 2002,pp. 25–28.

[14] D. McGregor, A. Kapolka, M. Zyda, D. Brutzman, Requirementsfor large-scale networked virtual environments, in: Proceedings ofthe Seventh International Conference on Telecommunications, June2003.

[15] N4264, Overview of MPEG-4 Standard, V.18—Singapore Version,2001.

[16] S. Ratnasamy, P. Francis, M. Handley, R. Karp, S. Schenker,A scalable content-addressable network, Proceedings of the 2001Conference on Applications, Technologies, Architectures, andProtocols for Computer Communications, pp. 161–172, 2001.

[17] M. Ripeanu, A. Iamnitchi, I. Foster, Mapping the Gnutella network,IEEE Internet Comput. January–February (2002).

[18] J. Ritter, Why Gnutella can’t scale, 2001. Available inhttp://www.darkridge.com/∼jpr5/doc/gnutella.html (Consulted onJune 2003).

[19] A. Rowstron, P. Druschel, Pastry: scalable, distributed object locationand routing for large-scale peer-to-peer systems, InternationalConference on Distributed Systems Platforms (Middleware),November 2001.

[20] D. Schoder, K. FischBach, Peer-to-peer prospects, Comm. ACM 46(2) (2003) 27–29.

[21] H. Shun-Yun, L. Guan-Ming, Scalable peer-to-peer networked virtualenvironment, Network and system support for games, Proceedingsof ACM SIGCOMM 2004 Workshops on NetGames ’04: Networkand System Support for Games, 2004, pp. 129–133.

[22] A. Singla, C. Rohrs, Ultrapeers: Another Step Towards GnutellaScalability, 2003, http://rfd-gnutella.sourceforge.net/Proposals/Ultrapeer/Ultrapeers.htm

[23] I. Stoica, R. Morris, D. Karger, F. Kaashoek, H. Balakrishnan,Chord: a peer-to-peer lookup service for internet applications, ACMSIGCOMM, 2001.

[24] W4415. ISO/IEC JTCL/SC29/WG11–N4415–MPEG-4 Systems,MPEG, ISO, Pattaya, December 2001.

[25] H. Zhang, A. Goel, R. Govindan, Incrementally improving lookuplatency in distributed hash table systems, Proceedings of the ACMSIGMETRICS Performance Evaluation Review, vol. 31, No. 1, 2003.

[26] B.Y. Zhao, J.D. Kubiatowicz, A.D. Joseph, Tapestry: an infrastructurefor fault-resilient wide-area location and routing, Technical ReportUCB/CSD-01-1141, U.C. Berkeley, April 2001.

Azzedine Boukercheis a Full Professor and holds a Canada ResearchChair position at the University of Ottawa. He is the Founding Director ofPARADISE Research Laboratory at Ottawa. Prior to this, he held a Fac-ulty position at the University of North Texas, USA, and he was workingas a Senior Scientist at the Simulation Sciences Division, Metron Corpo-ration located in San Diego. He was also employed as a Faculty at theSchool of Computer Science McGill University, and taught at Polytech-nic of Montreal. He spent a year at the JPL/NASA-California Institute ofTechnology where he contributed to a project centered about the spec-ification and verification of the software used to control interplanetaryspacecraft operated by JPL/NASA Laboratory. His research interests aredistributed simulation, collaborative virtual reality emergency prepared-ness, wireless multimedia and wireless ad hoc and sensor networks. Dr.A. Boukerche serves as a General Chair for the Eigth ACM/IEEE Sym-posium on modeling, analysis and simulation of wireless, and mobilesystems, and the ninth ACM/IEEE Symposium on distributed simulationand realt time-application. He also serves as an Associate Editor and ison the editorial board for ACM/Kluwer Wireless Networks, Wiley inter-national Journal of Wireless Communication and Mobile Computing, theJournal of Parallel and Distributed Computing, and the SCS Transactionson simulation.

Regina B. Araujo is adjunct professor at the Computer Science Depart-ment in Sao Carlos Federal University. Prof. Araujo has been involvedwith the creation and extensibility of collaborative virtual environments,through interactive non-linear stories since 2003, as well as the integra-tion of CVEs with wireless sensor networks for supervision and controlin the emergency preparedness application classes. Dr. regisna is a Pro-gram Co-Chair for the first ACM international workshop on Quality ofservice and Security for wireless networks.

Marcelo Laffranchi has a M.Sc. degree in computer science. His Maininterests are distributed collaborative virual environment.