A Scalable Adaptive Time Synchronization Protocolfor Large Scale Distributed Collaborative

Simulation Environment*Luqman Ahmad, Ming Zhang and Azzedine Boukerche

PARADISE Research LaboratorySchool of Information Technology and Engineering

University of Ottawa, Canada{lahma016,mizhang,boukerch}@site.uottawa.ca

Abstract—With the emergence of Massive Multiplayer OnlineGames (MMOG) and distributed distant learning/e-Learningapplications, Distributed Virtual Environments (DVE) have re-ceived a good deal of attention. As a highly human-computerinteracting environment, DVE provides an ideal platform for real-time message exchanges among virtual objects, human-beings.However, consistency and scalability are becoming indispensablechallenges considering the complexities of the existing heteroge-neous network architecture as well as the state synchronizationrequirement of DVEs. In this paper, we made an effort to addressthese issues by proposing a novel JXTA-based overlay peerto peer architecture for Large Scale Distributed CollaborativeVirtual Simulation Environments. Compared with our previousimplementations based on Department of Defense (DoD’s) HighLevel Architecture (HLA/RTI), this novel architecture is ableto provide consistency, scalability and fault tolerance by takingadvantages of state-of-the-art of the peer to peer computingparadigm. Our experimental results show that the scalabilityof DVEs is improved significantly whit our Peer-to-Peer DVEarchitecture. As a step further, we also propose a time and statesynchronization algorithm and evaluate its performance using aseries of simulation experiments.

I. INTRODUCTION

With the advances of multimedia communication and dis-tributed computing in recent years, Distributed Virtual Envi-ronment (DVE) is becoming a fundamental technique due tothe prevalence of e-Learning and multi-player games[5][3].In a DVE, the virtual worlds are shared amongst collocatedand remote users, and all the virtual environments must beable to see the effects of the real-time interactions at thesame time. In order to provide consistency in the virtualenvironments, all users should be able to perceive thesechanges in real time if any object triggers an event or anavatar changes its position. As a matter of fact, most of theexisting network architectures in DVEs and CVEs are basedon extended client/server architecture, in which the control ofthe whole simulation relies on the server or master nodes. Theclients in these DVEs update or modify their respective virtualenvironments and send update messages to the server, whichin return propagates the update messages to a group of clients

* This work is partially sponsored by Grants from NSERC Canada ResearchChair Program, Ontario Distinguished Researcher Award and Early CarrierResearch Award

in the network using IP-multicast technique [6]. However,in these architectures, the server needs efficient schedulingtechniques and becomes gradually overloaded as new clientsjoin the simulation. Therefore, adding new clients or avatarsincreases the workload of the server as well as the networktraffic, which eventually increases the network latency andinconsistency in the virtual environments. Therefore, the tradi-tional client/server architectures for the DVEs and CVEs scalewell due to the limitation of server resources and bandwidthconsumption. To increase the scalability, several approacheshas been proposed based on Peer to Peer (P2P) architecturethat achieved high scalability with large number of users bydistributing the load on the peers instead of servers[7][9][8].One of the key problems in the DVEs and CVEs is toensure scalability and consistency in the virtual environments.The scalability problem in this paper refers to the ability ofDVEs to accommodate a large number of concurrent userswithout sufficiently disregarding and encompassing the levelof responsiveness in order to provide real time interactiveexperience amongst the users. Whereas, the consistency refersto the topology consistency and state consistency that ensuresthat all the distributed virtual environments have same view.However, we have not focused on event consistency of theDVEs and are out of the scope of our paper. In this paper,we propose a novel scalable and state synchronized schemefor DVEs and CVEs which is implemented with the hierarchalservice oriented JXTA-core multi-layered architecture for largescale distributed simulations proposed in A. Boukerche et.al[1]. In our previous work [2] we have implemented our testbed application with High Level Architecture (HLA) whichwas developed by the DoD which have been a mature anddominant large scale distributed simulation standard for overa decade[10]. We have learnt that the High Level Architecture(HLA) has several demerits due to its nature of client/serverand heavy reliance on the Run Time Infrastructure (RTI)which makes HLA less scalable for Large Scale DistributedVirtual Environments (LSDVE) and non fault tolerant andresilient[1]. In order to enhance the capabilities of HLA/RTIa lot research has been conducted in this domain to improvethe scalability and interoperability of HLA/RTI and provide aService Oriented Architecture (SOA) such as eXtended Model-ing and Simulation Framework (XMSF)[4]. However, keeping

HAVE 2008 – IEEE International Workshop on Haptic Audio Visual Environments and their Applications Ottawa – Canada, 18-19 October 2008

978-1-4244-2669-0/08/$25.00 ©2008 IEEE

in mind the demerits of the other proposed architectures inthe light of DVEs and CVEs our proposed architecture isbased on a hierarchal service oriented JXTA-core multilayeredarchitecture for large scale distributed virtual environments.The reminder of our paper is organized as follows; Section IIwill discuss the background and related work, Section III willpresent our proposed scalable scheme along with our proposedscalable protocols, Section IV will present our implementationsetup and experimental results of our design and HLA/RTI(IEEE 1516) [2][3]. Section V will provide our conclusionand future work, followed by references.

II. BACKGROUND AND RELATED WORK

A lot of work and research has been conducted in enablingto cope with the scalability and consistency issues in DVEsand CVEs. Therefore, this section will provide a completepicture of the current and related efforts towards solving thescalability and consistency problems in Distributed VirtualEnvironments. Most of the efforts and approaches that havebeen proposed to cope with scalability and consistency fallsinto two categories; increasing the number of resources orreducing the consumption of bandwidth and update messages.

A. Increase Resources

In the first category, most of the popular approaches allowmultiple servers to host multiple worlds or deploy server-cluster to maintain a single world [11][13]. The butterfly Gridapproach on the other hand allows multiple servers for thesame game and each server servers a pre-determined maximumnumber of users. However, any attempt to join the server whenthe server is full is simply denied and users between differentservers may not interact with each other. T.A Funkhouser [4]proposed a server-cluster scheme that divides the single virtualenvironment in to zones and regions. This approach uses thenotion of server-based visibility that computes the potentialvisual interactions among the users in a single virtual envi-ronment and provides good and efficient mechanism for scal-able client/server systems. However, server-cluster approachis costly in terms of server side bandwidth consumptions,maintainability and hardware. SIMNET (simulator network-ing) [5] developed jointly a large scale distributed virtualenvironment with U.S. Army and Defense Advance ResearchProject Agency (DARPA). The aim of SIMNET was to createlarge scale low cost simulators suitable for military training.SIMNET architecture was client/server architecture and wasset up in a LAN environment in which each node broadcastevent updates to all other nodes. However, in the case oflarge number of users the message complexity increases toO (n) which makes the SIMNET scalable expensive in termsof bandwidth consumption

B. Reduce Bandwidth Consumption

The essence of this approach lies in the Interest Man-agement [3]. In order to deal with this problem InterestManagement uses the notion of Sphere of Interest (SOI).TheSOI acts as ,message filter that sends the relevant update

messages to the subscribed users. DVEs earlier than interestmanagement were communicating among the users by sendingbroadcast messages in the same LAN [5]. NPSNET [12] wasanother effort by U.S Naval Postgraduate School (NPS) whosegoal was to develop a military training DVE. NPSNET covereddifferent aspects of the DVE such as scalability, extensibility,interoperability and visualization. This approach proposed amulticasting filtering mechanism which was the building blockfor Interest Management latter [3]. To address the limitation ofserver side network resource by T.A Funkhouser [6], Diot etal [16] proposed a multicast distributed architecture MiMaze.The main contribution in MiMaze was not to address thescalability but to maintain the event consistency in the dis-tributed virtual environment by using a bucket synchronizationtechnique. However, the MiMaze improved the scalability andresponsiveness of the DVE. A simple Peer to Peer Voronoi-based Overlay Network (9) covered most of the scalabilityissues in the distributed Network Virtual Environment (NVE)[9]. It uses Voronoi diagram in order to construct and dis-tributed the virtual world in Voronoi cells. [9] claims thatit requires maintaining bounded bandwidth consumption rateat each peer which is quite impractical to justify. Anotherpowerful and simple scalable multicast communication pro-tocol SCORES [15], support multiple multicast groups. Themulticast groups in this approach dynamically partition thevirtual world into several spatial areas for state managementof the virtual environment

The distributed virtual environment DVEs, distribute thevirtual world among collocated and remote nodes. In orderto communicate and provide scalability DVE uses the afore-mentioned approaches of; increasing resources and decreasingbandwidth consumption. Most of the previous work discussedeither reducing the bandwidth consumption or increasing theresources. However, in our paper, we proposed hybrid Peerto Peer architecture based on increasing the resourcing byproviding a novel approach for running the DVE over clusterand GRID architectures and at the same time reducing resourceby reducing the number of update messages M, and bandwidthconsumption B. To the best of our knowledge, no other workhas focused on taking into account these indispensible factorsin their schemes which improves the scalability drastically ascompared to the previous work in this domain.

III. PROPOSED SCALABLE AND STATE SYNCHRONIZEDDESIGN

In this section we present our proposed scalable P2P schemefor DVEs and CVEs based on JXTA overlay network[11].Firstly, we will present the preliminaries of our proposedscheme and the assumptions we impose in our proposedarchitecture which will be followed by the overview of ourproposed system design. The remainder of the section willpresent our Message Passing and Control Service along withour joining and update protocols

A. Scalable and State Synchronized Design

As illustrated in Figure 1, each cluster has two local headsrunning our key services that ensure the real time message

passing and topology consistency in our DVE. Each peerimplements a local message compression service (LCS) whichenables the peer to compress and decompress the messagesbased on a predefined threshold value of the update messages.Similarly, the global message compression service (GMCS) isresponsible for compression and decompression of messagesexchanged in the global virtual cluster. In addition, GlobalMessage Passing and Control and Global Consistency Man-ager (GMC) services ensure the real time message passing andglobal topological consistency of our DVE. In order to focusand emphasize on the scalability and consistency problemswe have simplified our virtual environment which reflects ourprevious implementation of 3D Virtual Hospital (3DVH)[2].Our virtual environment is a synchronized distributed virtualenvironment in which events are modeled with discrete time-steps

Furthermore, our VE is mapped to 2D coordinate spaceresides on the local head peers with the help of Voronoidiagrams. Each waypoint constructs a Voronoi region. Weassume there are already optimal algorithms for constructingVoronoi diagrams with O(n log n) time complexity[15]. Themapping of the VE is done by Message Routing Service(MRS). The Prediction Vector Service (PVS) plays a vitalrole to send the update messages to the far neighbors in theVE by calculating the possible movement of updating avatar(the avatar that is changing the position) and Prediction TimeService (PTS) predicts the waiting time for each communi-cation link between local head and peers in the clusters, inorder to provide adaptability for synchronization. The PTSalso computes the frequency of number of update messagesfor the avatars farthest is in the DVE.

Fig. 1: Proposed Design for DVEs over GRID

B. Message Passing and Control Service Mechanism

The motivation behind our proposed P2P message passingtechnique was to provide a scalable architecture for distributedvirtual environments that is more suitable for both low andhigh bandwidth communication channels and consume lessnetwork resources. This has been achieved by introducing ourhybrid approach of using interest management and messagecompression techniques. The Message Passing and ControlService (MPCS) is the core of our proposed architecturethat uses the Message Routing Service (MRS) as a keyrouting scheme to propagate the relevant update messages tothe corresponding avatars in the VE. The MRS scheme isbased on matured and well studied mathematically constructVoronoi diagram [19].The aim of MRS is to provide a routingscheme mechanism in our proposed system design. We havechosen n waypoints in our DVE in the 2D plane; Voronoidiagram is constructed by partitioning the 2D space into nnon-overlapping cells in such a way that each cell containsone waypoint. Therefore, the entire 2D plane of our virtualworld is divided into arbitrary size in a deterministic manner.As illustrated in figure 2, each waypoint in the 2D coordinatemap construct and maintain a Voronoi diagram, based onthe spatial coordinates of the neighboring waypoints. Eachwaypoint contains a vector of its (x,y) coordinates, state andthe nearest neighbors positions list. Each avatar in the VE hastheir fixed Area of Interest (AoI) and a focus set, respectively.Each Voronoi region is marked as colored if the AoI of anavatar interests the Voronoi region as shown in the figure 2.The update messages that are exchanged among the avatarsare the position change messages. When the avatar in the VEchanges the position it sends the new position to the localhead peer. The local head peer computes shortest path to thecorresponding avatars in the virtual environment by using thewaypoints and Voronoi regions. The local head peer maintainsa counter CHop for each peer in the cluster. The CHop keepstrack of number of hops to reach from one avatar to anotherin the 2D-coordinate map and also keeps track of the largestand the smallest value of CHop for all the peers in the cluster.To compute the CHop values for an avatar, the local headtraverse the nearest waypoint based on the current positionof the avatar. Upon traversing, each waypoint marks its stateas visited and directs the local head to its nearest waypointby searching in its neighbors list and so on until it reaches aparticular avatar. The local head peer does not compute theshortest distance by using ways points if the following criteriaare met: the AoI of the updating avatar intersects the Voronoiregion of the nearest waypoint of the corresponding avatars,the focus set of the updating avatar intersect the Voronoi regionof adjacent avatars and the focus set of other avatar intersectsthe AoI or the Voronoi region of the updating avatar. In thesecases the local head peer will propagate every update messageto the adjacent avatar peer.

Based on the computed CHop values of the avatars, thelocal head peer maintains two multicast trees, Mmax andMmin for its cluster.The Mmax peers receive update messagesless frequently and periodically based on the Prediction TimeService(PTS)whereas, the Mmax peers receive every update

Fig. 2: 2D Coordinate Space of 3DVH with Voronoi Regions

message Due to network jitters, the round trip time amongthe peer and the local head peer varies. In order to achieveoptimal state synchronization and topological consistent vir-tual environments, we believe that it is very important tocompute the waiting time between the peers and local headpeers. As illustrated in the algorithm 1 and algorithm 2:Joining and Update Procedure, we have introduced the statesynchronization is achieved by setting the local clocks Lclkto the Lsync. The Lsync is the local head peer clock throughwhich the network delays and round trip time is computed inorder to synchronized the cluster communication. Similarly,the global clock resides at the global head peers that ensuresthe overall state synchronization of the global virtual cluster.This is done by Prediction Time Service (PTS) both in clustersand global virtual cluster. Every local head peer implementsthe PTS, which maintains a queue Qpid, with p number ofround trips. Our PTS is an adaptive approach that computesthe waiting time based on the previous round trip times in thequeue Qpid . Initially, ∆ is stored in Qpid which computedupon joining process of the peer. The round trip times isexponential random variable, and then the average waitingtime W(t) can be computed as:

W (t) = λmean =

p∑i=0

λi

N(1)

The arrival time is represented as λcurr, and the of historythe arrival times or delays λi is stored in theQpid. For moreaccurate waiting time results we have taken i = 1, 2, 3,...150,for our implementation. Let λmean be the computed time byabove equation and λcurr is the recent delay received by thelocal peer head, then if the λmean < λcurr then we expressthe waiting time W(t) in terms of last waiting time β(t),

W (t) = (λcurr − λmean) + β(t) (2)

In addition, the messages are routed to the Mmax multicasttree less frequently as compared to the Mmin multicast treenodes D. The frequency of propagating the update messagesto the Mmax can be computed by taking the average waitingtime taken by all the peers that are child in Mmax.

Algorithm 1 Join and Update Peer Side Procedure

Initialize: Lclk ⇐ 0Pos⇐ getPosition()Gclk ⇐ getSystemTime()Tsend ⇐ Gclkif Trigger then

send(Msg:Joinrequest, Pid, Cid Tsend, AoI, Pos )end ifloop

if received message is joined thenTδ ← Gclk − TαTγ ← Tδ − TαTγ ← Tδ + LsyncLclk ← Tγ∆← Lclk − LsyncstartLocalThread(Lclk)M ← (joined,∆, Lclk)if checkSize(M) > threshold then

send (compress(M)) to local headelse

send(M)end if

else if Peer is updating peer Pupdate thenM ← (Pos, Lclk, speed(v),Θnew)if checkSize(M) > threashold then

send (compress(M)) to local head peerelse

send (M) to local head peerend if

else if update Scene then{Perform compression check }∆← Lclk − Lsyncif M is in Qlocal then

send(NACK) to local head peerelsePosnew ← Posθnew ← αQLocal ←Msend(ACK,∆, Lclk)

end ifelse

TERMINATE {Joined request declined}end if

end loop

F (t) =

G∑i=0

λmean

G(3)

In order to provide a consistent view of all the distributedvirtual environments in the simulation, our Prediction VectorService (PVS) send the possible motion of the updating avatarto the avatars of Mmax multicast tree. The PVS is basedon the speed v, time t, position (x,z) and angle Θ of focusset. We have restricted the focus set angleθ < π

2 , becausewe believe that the avatar will change its position within

Algorithm 2 Join and Update Procedure Local Head Peer Side

Initialize: Lsync ← CHop ← 0Trec ← getSystemTime()loop

if M is joined then∆← Trec − TsendQPid ← ∆{call computerWay(new peer) fucntion}

else if M is ACK or NACK then∆← ∆ + Lsync − LclkQPid ← ∆{Increment the G counter if in Mmax else D counter}{Stop waiting}

else if M is update then{Perform Compression check}∆← ∆ + Lsync − LclkQPid ← ∆if Update message from Controlling Pcon then

call computeWay()send update Message to Mmin peersCheck F(t) value then send to Mmax peers if F(t)expires

else{the update message is piggyback message}send M to Pcon

end ifend if

end loopFunction: computeWay()if argument is new in the function then

MarkVoronoiReg(AoI) {Construct Voronoi Rgion wrtAoI}loopRegioncon ← CheckV oronoiRegion(){check Voronoi region of waypoint}Interestcon ← CheckAoI()if Regioncon OR Interestcon then{Check Minimum hops}if CHop ≤MinHop then

insertMmin(Pid)N ← N + 1D ← D + 1joinCluster(Pid)send (M ← joined, Tα, Lsync)

elseinsertMmax(Pid)N ← N + 1G← G+ 1joinCluster(Pid)send (M ← joined, Tα, Lsync)

end ifelse{ Traverse near neighbor Nnearest }CHop ← CHop + 1

end ifend loop

end if

Fig. 3: Possible Movement of Avatar

π2 . The new angle computed Θnew is based on the rotationangle α and the possible angular motion of avatar Θnew.Upon receiving the prediction vector the peer will calculatethe current angle and position coordinates and offset the avatarangle with the difference of Θnew the current rotation angle α.The Consistency Manager (CM) ensures the local consistencyof the cluster. It ensures the topological consistency by keepingtrack of all the ACK and NACK messages of Mmin and Mmax

of the peers. The avatars which are member of the Mmax

multicast tree send the ACK messages to their correspondinglocal head peers whereas, the avatars which are near theupdating avatar send their positions to the local head peer withpiggybacking the ACK signals in their update messages. Uponreceiving the updates piggyback messages the local head peer,routes the position of the near avatars to the current updatingavatar in the virtual environment. The current updating avatarupdates its virtual environment with respect to the receivedupdates of the near avatars. Thereafter, the local head peerensures the it has received the G-1, where G-1 is the numberof Mmax , ACK messages from the Mmax and D -1 updatemessages from the Mmin peers. Upon receiving the updateACK messages and the update messages from Mmin, the localhead peer broadcasts the ACKCid messages to the globalvirtual cluster.

IV. IMPLEMENTATION SETUP ANDEXPERIMENTAL RESULTS

To evaluate the robustness and performance of our designand algorithm, we implement a 3DVH distributed VirtualEnvironment please refer to [2] for virtual environments.The underlying technology we used in our implementationis Sun Microsystems JXTA-core [11] overlay networks. Thehardware platforms used in our experiments is made up of12 computers of 2.8 GHz processors and 2 MB RAM each.For high performance graphics each peer has Nvidia G73GPU with 512 MB video RAM . A standard 100Mbps fastEthernet switch interconnects them. Our test environment is(300 x 150) with five avatars and each avatar has its own localview. To validate our design and algorithm for message passingand consistency, we decided to investigate two scenarios withHLA/RTI (Client/Server) and with our design (P2P).

Figure 4, shows the average number of messages exchangeswhen we run the simulation. We can conclude that HLA/RTI

Fig. 4: Bandwidth Consumption

Fig. 5: Bandwidth Consumption

propagates more messages as compared to our other designsdue to client/server and static public/subscribe nature. JXTA-P2P design, without our PTS mechanism exchanges moremessages then with PTS mechanism. The reason is due to fixedamount of waiting interval for each communication link. Onthe other hand our adaptive synchronization technique (JXTA-PTS)reduces substantial amount of messages. In addition,figure 5, shows the bandwidth consumption of our techniquewhich is much less as compared to HLA/RTI and basic JXTA-P2P architecture. On the Contrary, figure 6 shows the overalltopological consistency remains in the range of 70 % to 90%. Thus, our experimental results provide scalable approachby reducing the overall bandwidth consumption and providesa state synchronized simulation framework for large scaledistributed Simuluations/DVEs.

Fig. 6: Overall Topological Consistency

V. CONCLUSION AND FUTURE WORK

In this paper we presented our scalable P2P design fordistributed virtual environments and addressed the scalabilityissues in large scale distributed virtual environments. Due

to the fact the heterogeneity in P2P is quite extreme, wedesigned our P2P DVE architecture on JXTA-CORE overlaynetwork that provides SOA functionalities. We believe ourdesign is most suitable for Cluster and GRID based distributedsimulations and virtual environments that faces the scalabilityproblems. Preliminary results, from our experiment show thatour design and algorithms provides potential breakthrough forscalability and consistency issues in the DVEs and CVEs onlarge scale distributed Virtual Environment. Furthermore, inour design we have not been taken in account the networkdelay, CPU processing and event consistency. Our future workwill address these issues in details in the light of our proposeddesign.

REFERENCES

[1] Azzedine Boukerche and Ming Zhang, Towards Peer-to-Peer Based Dis-tributed Simulations on a Grid Infrastructurem, 41st Annual SimulationSymposium,pp:212-219, 2008.

[2] L. Ahmad, A. Boukerche, A. Al Hamidi, A. Shadid, R. Pazzi, Web-based e-learning in 3D large scale distributed interactive simulationsusing HLA/RTI, IEEE International Parallel and Distributed Processing,pp:6-10,2008.

[3] Morse, K.L, Pullen Mark and Tolk A, An Architecture for Web-ServicesBased Interest Management in Real Time Distributed Simulation, Dis-tributed Simulation and Real-Time Applications,pp: 108-115, 2004

[4] D. C. Miller and J. A. Thorpe, SIMNET: The Advent of SimulatorNetworking, Proc. IEEE, vol. 83, pp:1114-1123, Aug. 1995

[5] Jean-Sebastien B, Jorg K, Comparing Interest Management Algorithmsfor Massively Multiplayer Game, Proceedings of 5th ACM SIGCOMMworkshop on Network and system support for games (NSSG’06), pp:24-32,2006.

[6] T. A. Funkhouser, RING: A client-server system for multi-user virtualenvironments Symp. Interactive 3D Graphics, pp: 85- 95,1995.

[7] Thorsten H,Thomas B, Robert H, A peer-to-peer architecture for massivemultiplayer online games, 5th ACM SIGCOMM workshop on Networkand system support for games, pp:34-42, 2006.

[8] A. E. Saddik and A Dufour, Peer to Peer Suitability for Collaborative Mul-tiplayer Games, Seventh IEEE International Symposium on DistributedSimulations and Real-Time Applications, pp:101-109, 2003

[9] Shun-Yun, Guan-Ming, Scalable Peer-to-Peer Networked Virtual Environ-ment, Proceedings of 3rd ACM SIGCOMM workshop on Network andsystem support for games

[10] S. Singhal and M. Zyda, Networked Virtual Environments: Design andImplementation, ACM Press, New York, 1999.

[11] JXTA, JXTA Java Standard Edition v 2.5 Programmers Guides, 2007.[12] Lety, E.; Turletti, T.; Baccelli, F., SCORE: a scalable communication

protocol for large-scale virtual environments ,Networking, IEEE/ACMTransactions, vol. 12, pp: 247-260, 2004 .

[13] C. Diot, L. Gautier, A distributed architecture for multiplayer interactiveapplications on the Internet, IEEE Network, vol. 13, pp: 6-15, 1999.

[14] T. Mldner, G. Leighton, J. Diamond, TREECHOP: A Tree-based Query-able Compressor for XML, IThe Ninth Canadian Workshop on Informa-tion Theory, 2005.

[15] L. Guibas and J. Stolfi, Primitives for the manipulation of generalsubdivisions and the computation of Voronoi diagrams, ACM Trans.Graphics, vol. 4, pp:74-123, April 1985

[16] Zhou S, Cai W, Lee B, Turner, S, Time-Space Consistency in Large ScaleDistributed Environments, ACM Trans on Computational Logic Vol.V,pp:1-17, 2003.

[17] Lothar P, Lars C.W, On the suitability of dead reckoning schemes forgames, Proceedings of the 1st workshop on Network and system supportfor games (NSSG’02) pp:79 - 84 , 2002

[18] Takahiro Y,Yutaka I, Tomohito I, Influences of network latency andpacket loss on consistency in networked racing games, Proceedings of 4thACM SIGCOMM workshop on Network and system support for games,pp:1-8, 2005