VON: A Scalable Peer-to-Peer Network for Virtual Environments

1

VON:A Scalable Peer-to-Peer Network

for Virtual Environments

Shun-Yun Hu (胡舜元 ) ([email protected])

CSIE, National Central University, Taiwan

2005/10/19

2

Outline

Introduction Voronoi-based Overlay Network (VON) Simulation Results Analysis Conclusion

3

What is Networked Virtual Environment (NVE)?

Virtual Reality + Internet

3D environment with people (avatar), objects, terrain, agents

Military simulations (’80) Massively Multiplayer Online Games (mid-‘90)

Trends: larger scale, more realistic simulation

4

5

NVE: A Shared Space

6

Issues for Creating NVE

Consistency (events/states)

Responsiveness multiplayer Security

Scalability Persistency massively multiplayer Reliability (Fault-tolerance)

7

The Scalability Problem

Many nodes on a 2D plane ( > 1,000) Message exchange with those within Area of Interest (AOI) How does each node receive the relevant messages?

Area of Interest

8

A simple solution (point-to-point)

N * (N-1) connections ≈ O(N2) Not scalable!

Source: [Funkhouser95]

9

A better solution (client-server)

Message filtering at server to reduce trafficN connections = O(N) server is bottleneck


10

Current solution (server-cluster)

Still limited by servers. Expansive to deploy & maintain.


11

Scalability Analysis

Scalability constrains Computing resource (CPU) Network resource (Bandwidth)

Non-scalable system vs. Scalable system

x: number of entities

y: resource consumption at the limiting system component

Resource limit

12

Server-cluster issues

Insufficient total resourceHardware provisioning over-provision!

High user density (crowding)User limits limits scale & realism!

Excessive inter-server communicationsLess load balancing difficult balance!

13

What Next?

Strategies Increase resource More servers Decrease consumption Message filtering

Architectures Scale Point-to-point (LAN) tens 10^1 Client-server hundreds 10^2 Server-cluster thousands 10^3 ? millions 10^6 …

Peer-to-Peer

14

What is Peer-to-Peer (P2P)?

[Stoica et al. 2003] Distributed systems without any centralized control

or hierarchical organization Runs software with equivalent functionality

Examples File-sharing: Napster, Gnutella, eDonkey Distributed Search: Chord, CAN, Tapestry, Pastry VoIP: Skype

15

Peer-to-Peer Overlay

A P2P overlay network source: [Keller & Simon 2003]

16

Promise & Challenge of P2P

Promises Growing resource, decentralized

Scalable Commodity hardware Affordable

Challenges Topology maintenance dynamic join/leave Efficient content retrieval no global knowledge

17

Issues for Creating P2P NVE

Consistency (events/states)

Responsiveness multiplayer Security

Scalability Persistency massively multiplayer Reliability (Fault-tolerance)

Consistency (topology) P2P NVE

18

Related Work (1):

DHT-based: SimMUD

[Knutsson et al. 2004] (UPenn)

Pastry + Scribe Regions Coordinators

(super-nodes)

Fixed-size region Relay overhead

19

Related Work (2):

Neighbor-list Exchange

[Kawahara et al. 2004] (Univ. of Tokyo)

Fully-distributed Nearest-neighbors List exchange

High transmission Overlay partition

20

Related Work (3):

Mutual Notification: Solipsis

[Keller & Simon 2003] (France Telecomm R&D)

Links with AOI neighbor Mutual cooperation Inside convex hull

Potentially slow discovery Inconsistent topology

21

Outline


22

Design Goals

Observation: for virtual environment applications, the contents we want

are messages from AOI neighbors Content discovery is a neighbor discovery problem

Solve the Neighbor Discovery Problem in a fully-distributed, message-efficient manner.

Specific goals: Scalable Limit & minimize message traffics Responsive Direct connection with AOI neighbors

23

Voronoi Diagram

2D Plane partitioned into regions by sites, each region contains all the points closest to its site

Can be used to find k-nearest neighbor easily

Neighbors

Site

Region

24

Design Concepts

Identify enclosing and boundary neighbors Each node constructs a Voronoi of its neighbors Enclosing neighbors are minimally maintained Mutual collaboration in neighbor discovery

Circle Area of Interest (AOI)

White self

Yellow enclosing neighbor (E.N.)

L. Blue boundary neighbor (B.N.)

Pink E.N. & B.N.

Green AOI neighbor

L. Green unknown neighbor

Use Voronoi to solve the neighbor discovery problem

25

Procedure (JOIN)

1) Joining node sends coordinates to any existing node

Join request is forwarded to acceptor

2) Acceptor sends back its own neighbor list

joining node connects with other nodes on the list

Acceptor’s region Joining node

26

Procedure (MOVE)

1) Positions sent to all neighbors, mark messages to B.N.

B.N. checks for overlaps between mover’s AOI and its E.N.

2) Connect to new nodes upon notification by B.N.

Disconnect any non-overlapped neighbor

Boundary neighbors

New neighbors

Non-overlapped neighbors

27

Procedure (LEAVE)

1) Simply disconnect

2) Others then update their Voronoi

new B.N. is discovered via existing B.N.

Leaving node (also a B.N.) New boundary neighbor

28

Dynamic AOI

Crowding within AOI can overload a particular node

It’s better if AOI-radius can be adjusted in real time

29

Adjustment Conditions

AOI-radius decrease Number of connections > connection limits

AOI-radius increase Maximum connections not exceeded Current AOI-radius < preferred AOI-radius

Mutual awareness rule Do not disconnect a neighbor who sees me

30

Demonstration

Simulation video General movements (40 nodes, 800x600 world) Local vs. global view Dynamic AOI adjustment

31

Outline


32

Simulation Method C++ implementation of VON (open source VAST library)

World size: 1200 x 1200, AOI: 100 Trials from 100 – 1000 nodes Connection limit per node: 20 1000 time-steps

(~ 100 simulated seconds, assuming 10 updates/seconds)

Behavior model Random movement: random destination Constant velocity: 5 units/step Movement duration: random (until destination is reached)

33

Consistency Metrics

Topology Consistency [Kawahara, 2004]

observed AOI neighbors

actual AOI neighbors

Drift Distance [Diot, 1999]Distance between observed position and actual position

(average over all nodes)

34

Scalability: Avg. Transmission / sec

0

2

4

6

8

10

12

14

16

0 200 400 600 800 1000Number of Nodes

Siz

e (

kb

)

basic

dAOI

basic (fixed density after 500 nodes)dAOI (fixed density after 500 nodes)

35

Scalability: Max. Transmission / sec

0

10

20

30

40

50

60

0 200 400 600 800 1000Number of Nodes

Siz

e (

kb

)

basic

dAOI

basic (fixed density after 500 nodes)dAOI (fixed density after 500 nodes)

36

Scalability: Avg. Neighbor Size

0

5

10

15

20

25

30

35

40

45

50

0 200 400 600 800 1000Number of Nodes

Ne

igh

bo

r S

ize

connected neighbors (basic)

AOI neighbors (basic)

connected neighbors (dAOI)

AOI neighbors (dAOI)

37

Consistency: Topology Consistency

99.90

99.91

99.92

99.93

99.94

99.95

99.96

99.97

99.98

99.99

100.00

100 200 300 400 500 600 700 800 900 1000Number of Nodes

To

po

log

y C

on

sis

ten

cy

(%

)

basicdAOI

38

Consistency: Drift Distance

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

100 200 300 400 500 600 700 800 900 1000

Number of Nodes

Ave

rag

e D

rift

Dis

tan

ce

basic

dAOI

39

Reliability: Effects of Packet Loss

0

10

20

30

40

50

60

70

80

90

100

0% 20% 40% 60% 80% 100%Loss Rate

Un

its

Topology Consistency (%)

Recovery Steps

40

Outline


41

Analysis of Design

Scalability Bounded resource consumption dynamic AOI

Consistency (Topology) Topology is fully connected enclosing neighbors

Reliability Self-organizing distributed neighbor discovery

Responsiveness Lowest latency direct connection, no relay

42

P2P NVE Comparisons

DHT-based

Neighbor-list exchange

Solipsis VON

Consistency (topology)

DHT & Supernode

(consistent)

Neighbor list-exchange

(partitioning)

Neighbor notify&query

(undiscovery)

Neighbor notify

(consistent)

Responsive-ness

two to many

One hop One hop One hop

Scalability O(n) on supernode

Constant in crowding

Constant if fixed density

Constant in crowding

Con Latency too high

Overlay partitioning

Occasional undiscovery

Circular node line-up

43

Problems of Voronoi Approach

Message traffic Circular round-up of nodes Redundant message sending

(inherent to fully-distributed design)

Incomplete neighbor discovery Can happen with inconsistent / incorrect neighbor list Fast moving node

Limited AOI Direct connections

44

Outline


45

Summary

NVE scalability is achievable with P2P architecture and is a neighbor discovery problem

A promising solution: Voronoi-based P2P Overlay Leverage knowledge of each peer to maintain topology

Properties Scalable: fully-distributed, dynamic AOI Efficient: low irrelevant messages, zero relay Simple: simple protocol and procedure

46

Potential Applications

Online gamesPosition updates in current MMOGs, Voice-chats

MilitaryEnable large-scale, affordable military training simulation

3D WebProvide multi-user interactivity to static 3D world

Scientific simulationsDistribute spatial simulation requiring frequent synchronization

47

Future Perspectives

Short-term Distributed event/state consistency Customizable AOI (Heterogeneity in P2P) Recovery from overlay partition and fast-moving nodes

Long-term Persistency issue (P2P-based database) Security issue (protection from malicious nodes) 3D content distribution (3D streaming on P2P)

Massive, persistent 3D environment sharable by all!

48

Acknowledgements Dr. Jui-Fa Chen (陳瑞發老師 ) Tsu-Han Chen (鄭子涵 ) Members of the Alpha Lab, TKU CS

Dr. Chin-Kun Hu (胡進錕老師 ) Guan-Ming Liao (廖冠名 ) LSCP, Institute of Physics, Academia Sinica

Joaquin Keller (France Tele. R&D, Solipsis) Jon Watte (there.com) Kuan-Ta Chen (陳寬達 , NTU)

49

Q&A

Thank you!

[email protected]://vast.sourceforget.net

(http://vast.sf.net)

50

Inconsistency caused by dAOI

51

Topology Consistency in NLE

52

Slow Discovery in Solipsis

Documents

VON: A Scalable Peer-to-Peer Network for Virtual Environments