1

Scalable Peer-to-Peer Virtual Environments

Shun-Yun Hu

CSIE, National Central University, Taiwan

2008/06/03

2

4

Massively Multiplayer Online Games

MMOGs are growing quickly Multi-billion dollar industry 10 million subscribers for World of Warcraft 600,000 concurrent users, but 3,000 per world

Can we scale to millions in the same world?

Imagine you start with a globe

Zoom in…

To Trier…

and to Universitat Trier

Right now it’s flat…

But in the near future…

Virtual Environments (VEs): A shared space

14

Model for virtual worlds

Many nodes on a 2D plane Message exchange with those within Area of Interest (AOI) How does each node receive the relevant messages?

Area of Interest

15

A simple solution (point-to-point)

But…too many irrelevant messagesN * (N-1) connections ≈ O(N2) Not scalable!

Source: [Funkhouser95]

16

A better solution (client-server)

Message filtering at server to reduce trafficN connections = O(N) server is bottleneck

Source: [Funkhouser95]

17

Current solution (server-cluster)

Still limited by servers. Expensive to deploy & maintain.

Source: [Funkhouser95]

The Problem

Client-server: resources limited by provisioning

Resource limit

[Funkhouser95]

The Solution

Peer-to-Peer: resources grow with demand

Resource limit

[Keller & Simon 2003]

Outline

Overlay management (VON)

State management (VSM)

Client-assisted service (FLoD)

Voronoi-based Overlay Network

(VON)

Design Goals

Observation: for VEs, the contents are messages from AOI neighbors Content discovery is a neighbor discovery problem

Specific goals: Scalable Limit per-node message traffic Responsive Direct connection with AOI neighbors

23

If you talk with your AOI Neighbors directly…games can be built

But how to discover new neighbors?

24

Voronoi Diagram

2D Plane partitioned into regions by sites, each region contains all the points closest to its site

Can be used to find k-nearest neighbor easily

Neighbors

Site

Region

25

Design Concepts

Identify enclosing and boundary neighbors Enclosing neighbors are minimally maintained Boundary neighbors mutually help to discover neighbors

boundary neighbor (B.N.)L. Blue

unknown neighborL. Green

normal AOI neighborGreen

E.N. & B.N.Pink

enclosing neighbor (E.N.)

Yellow

selfWhite

Area of Interest (AOI)Circle

Use Voronoi to solve the neighbor discovery problem

26

Procedure (MOVE)

1) Positions sent to all neighbors, mark messages to B.N.B.N. checks for overlaps between mover’s AOI and its E.N.

2) Connect to new nodes upon notification by B.N. Disconnect any non-overlapped neighbors

Boundary neighbors

New neighbors

Non-overlapped neighbors

Dynamic AOI

Crowding within AOI can overload a particular node

It’s better if AOI-radius can be adjusted in real time

28

Demonstration

Simulation demo Random movements (100 nodes, 1200x700 world) Local vs. global view Dynamic AOI adjustment

Simulations C++ implementation of VON (open source VAST library)

World size: 1200 x 1200 (AOI: 100) Trials from 200 – 2000 nodes Connection limit: 20 3000 time-steps

(~ 300 simulated seconds, assuming 10 updates/seconds)

Behavior model Random movement: random waypoint Constant velocity: 5 units/step Movement duration: random (until destination is reached)

Scalability: Avg. Transmission / sec

0

5

10

15

20

25

30

0 400 800 1200 1600 2000Number of Nodes

Siz

e (kb /

basic

dAOI

basic (fixed density after 1000 nodes)dAOI (fixed density after 1000 nodes)

Scalability: Max. Transmission / sec

0

10

20

30

40

50

60

70

0 400 800 1200 1600 2000Number of Nodes

Siz

e (kb /

basic

dAOI

basic (fixed density after 1000 nodes)dAOI (fixed density after 1000 nodes)

Voronoi State Management

(VSM)

33

State Management for VEs

Besides positions, object states are important too

Three main issues: Consistency control Load balancing Fault tolerance

A basic approach

Let game states be managed by all clients Each client has two roles: peers & arbitrators i.e., Voronoi partitioning (we can use VON)

Problems with basic approach

O(n2) connections at hotspots Some cells have large sizes Constant ownership transfer

36

VSM: solution ideas

Connection overload → Aggregators clustering Large cell-size → Virtual peers incremental

transfer Constant transfers → Explicit ownership transfer

37

VSM: Consistency control

Managing arbitrator receives and processes events

Events are forwarded if necessary

Updates sent to affected arbitrators, then peers

38

VSM: Consistency control

Managing arbitrator receives and processes events

Events are forwarded if necessary

Updates sent to affected arbitrators, then peers

39

VSM: Consistency control

Managing arbitrator receives and processes events

Events are forwarded if necessary

Updates sent to affected arbitrators, then peers

40

VSM: Consistency control

Managing arbitrator receives and processes events

Events are forwarded if necessary

Updates sent to affected arbitrators, (then peers)

41

VSM: Load balancing

Traditional: high-capacity nodes first, then adjust VSM: low-capacity nodes first, then

cluster Overload: ask for aggregator, submit control Underload: disintegrate, release control

42

VSM: Fault tolerance

Regular arbitrator: Pick backup arbitrator, backup states Backup transfers ownership to enclosing arbitrators

Aggregators: Pick backup aggregators Take over original if failed Choose new backup

Peer-to-Peer 3D Streaming

44

Background

MMOGs today need DVD installations

Too slow and unpractical for: Larger and more dynamic worlds (TBs data) More numerous worlds (Web 3D)

Content streaming is needed 80% - 90% content is 3D (e.g., 3D

streaming)

3D streaming

Continuous and real-time delivery of 3D content to allow user interactions without a full download. base & refinement pieces

Base 1 2 3

Refinements

User

(Hoppe 96)

Scene streaming

Multiple objects Object determination & prioritization

[Teler &

Lischinski 2001]

47

3D streaming vs. media streaming

Video / audio media streaming is very matured

User access patterns are different for 3D content Highly interactive Latency-sensitive Behaviour-dependent Non-sequential

How to scale to millions of concurrent users?

48

Observation

Limited & predictable area of interest (AOI) Overlapped visibility = shared content

49

overlapped visibility = shared content

Challenges for P2P 3D streaming

Appropriate peer grouping Matching interests / needs Matching capabilities

Dynamic group management Interest groups are dynamic (non-sequential) Real-time constraints (latency-sensitive)

Minimal server involvement Visibility determination (object selection) Request prioritization (piece selection)

FLoD [Infocom 2008]

VE partitioned into cells with scene descriptions Assumes P2P overlay that provides AOI neighbors

star: self triangles: neighborscircle: AOI rectangles: objects

52

Neighbor discovery via VON

Boundary neighbors

New neighbors

Non-overlapped neighbors

[Hu et al. 06]

Voronoi diagrams identify boundary neighbors for neighbor discovery

53

Prototype experiment

Progressive models in a scene (by NTU) Peer-to-peer AOI neighbor requests (by NCU) Found matching client upload / download

54

Simulation setup

Environment 1000x1000 world, 100ms / step, 3000 steps client: 1 Mbps / 256 Kbps, server: 10 Mbps (both)

Objects Random object placement (500 objects) Object size based on prototype

User behavior Random & clustering movement (1.5 * ln(n) hotspots)

55

Server bandwidth usage

56

Client bandwidth usage

Impacts of P2P VEs…

No server as bottleneck scalable Commodity hardware affordable

2D web 3D web Earth-scale virtual worlds

(millions/billions of people)

Unresolved issues

Issues for creating VEs

Consistency Interactivity multiplayer Security

Scalability Persistency massively multiplayer Reliability

Interoperability Content

3D web

Meshing physical & virtual topologies

Client 2

Client 1

A generic pub/sub scenario

pub sub

62

Q&A

VON: A Scalable Peer-to-Peer Network for Virtual EnvironmentsIEEE Network, vol. 20, no. 4, Jul./Aug. 2006

FLoD: A Framework for Peer-to-Peer 3D StreamingIEEE INFOCOM, Apr. 2008

Thank you!

http://vast.sourceforge.nethttp://ascend.sourceforge.net

Unresolved issues

Overlay management Topology-aware, capacity-matching superpeers Flexible publication / subscription Direct vs. forwarding deliveries

State management Load balancing (high user density) Persistency Security

Client-assisted services (e.g., P2P 3D streaming) Source nodes discovery Visualization vs. networking priority matching LOD considerations

Other issues

Common API

Shared simulator / platform

Interoperability

Extension: VoroCast

Pack reduction via forwarding Headers reduction Data compression & aggregation