Bandwidth Reduction and Latency Tolerance in Real Time Internet Applications Multi-user Extensible Virtual Worlds 1

Bandwidth Reduction and Latency Tolerance in Real Time Internet Applications

Multi-user Extensible Virtual Worlds

1

Communication Optimizations

• Fast archiving process• Archives cached and shared with multiple clients

– Reduces overhead for many clients in same city

• Client-side transform interpolation– Allows reduced transmission frequency

• Offloaded work to the client– Generating deterministic assets during city load– Performing mesh animation locally

• Secure handshake authentication on connect

2

Previous Attempts

• Combining common-practice methods– Low-level optimizations– Caching and re-use– Interpolation– Prediction and smooth correction

3

• Allows server to send updates less frequently

• Maintains a smooth experience

Drawn to Screen

Positions From Server

Client-side interpolation

4

Primary Goals

• Latency tolerance beyond state of the art• Reduce bandwidth beyond state of the art

5

0

2

4

6

8

10

12

14

16

1000 2000 3000 4000 5000 6000 7000 8000 9000 10000

Tota

l Ser

ver B

andw

idth

(GB/

sec)

Player Count

Previous Attempts

• Not good enough for our scale– Massive activity imposes bandwidth and

synchronization burden beyond the norm• Need server to be further “forward in time”

for higher latency tolerance• Need to find a way to lower bandwidth

further– Simply reduce rate of update frames further?

6

Original Server Injection Model of Prediction

• Interpolation smooths movement until server stalls– Client interpolates based upon expected arrival time– If server increases lag, there is nothing to interpolate to!

• Inject copy of Server functionality into Client– Performs same work on subset of data for prediction

• Server state may differ from prediction– Client interpolates what user sees during correction

7

Client

Server

Server

Failure of Simple Injection Model

• Small differences produce large changes– Collision events, House construct selection

• Some states have large prediction failure consequences– Corrections become as dramatic as having

allowed video to “stutter”• Does not allow us to tolerate severe

internet latency in practice

8

Accomplished Work

• Established persistent server on the IBM z10 enterprise server running 24/7

• Lowered power consumption during low utilization

• Characterized latency across typical wired and wireless networks

9

Latency Characteristics

• Large latency due to software pipelines– Exacerbated by present interpolation system

• Intermittent network latency can triple shown value– Some data gathered from Internet Weather Map project

10

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

Loc WiFi Cnty CA USA Lndn Eur

NetworkPipeline

Accomplished Work

• Tested server-in-client injection model for latency tolerance– Results indicate infeasible approach to

increase time lag between server and client• Designed new synchronization

architecture to accomplish both latency tolerance and bandwidth reduction– Presently being implemented

11

New Synchronization Architecture

• Synchronization is a common problem– All multiplayer systems– Application-specific issues dominate problem

• Generalized solution for wider applicability– Can be applied to many client-server systems

• Unified design addresses both bandwidth and latency problems in real-time distributed applications

12


• States synchronized as separate streams– Server at different virtual time in all streams!

• Each stream can be at different clock• Client performs prediction and correction

only for streams the clock has overrun– Partial-knowledge prediction

• Each stream can be abstracted differently– Values, predictor inputs for values, etc.

13


• Decouples problems of each kind of state– Each stream poses different trade-offs

• Taxonomy of properties aids application• Exposes necessary model modifications

14


• State properties– Predictability (level of determinism)

• How can it be computed locally?

– Computational dependencies• What other states are required to compute locally?

– Error magnification effect• How do errors in my computational dependencies

magnify errors in locally computed state?

15


• Events given in terms of a virtual clock– Precise clock synchronization is impossible– Prevents errors from propagating forward– Virtual clock time is adjustable

• Accuracy implies all clients view environment accurately at some (relatively close) point in time

16

Example: Object Transforms

• Moving objects affected by forces– Assume no other effects for now– Physical processes accurately predictable

• Architecture & executable code may differ– Float point drift must still be compensated for– Intermittent object transform updates required

• Vastly improves latency & bandwidth– Only a potential due to real dependencies

17

Complication: Collisions

• Also predictable, but errors magnify– Dissimilar collision computations can

drastically affect future object transforms• Server notifies clients: collisions & misses

– Clients are notified of “close calls” that miss!– Server exists forward in time from clients

• Example of predictable computational dependency with error magnification

18

Animation: Buildings select and grab objects

• Objects “pulled into place” in structure• Current selection criteria based upon

proximity to building– Coupled to transforms: circular dependency!– Initially developed for selection efficiency– Not imperative, can be modified

• Errors have largest magnification yet!– Objects change subsystems, alter paths

19

Animation: Buildings select and grab objects

• Push selection further forward in time– Server notifies clients of selections– Reduces client need to predict selection

• Selection criteria modified– Allows server to compute future more easily

• Example of untenable error magnification– Simulation model altered to accommodate!

20

Last Example:Player Location

• Player cyclone applies forces to objects– Creates dependency with object transforms

• Problem: Clients are not “forward in time”– Player location not well predictable

• Solution: loosely couple wind forces from player visual representation– Make physics manifestation more predictable

21


• Common position predictor function utilized by server and all clients– Deterministic function defines position of

physical manifestation– Function uses intermittent player state as

input• Periodic client player velocity update

enough to maintain

22


• Still relatively low-latency: prediction failure results in compensation– Compensation should be soft (low error

magnification)• Example of modified simulation with low

error magnification– Reduces player position traffic by abstracting

to velocity functions

23

Synchronization Examples

Object Transforms

Object Collisions

Building Selection

Player Location

Predictability Good Good Modified towards Good

Modified towards Good

Dependencies All others listed

Object Transforms

None Player Input

Error Magnification

Small Medium Large Small to Medium

24

Where Are We On This?

• Design complete but undergoing iteration• Implementation underway

– No drift on test using Verlet integration kernel– Most work yet to be done

• Generalizations being developed– Applies to real-time internet applications

25

Summary

• Developing revolutionary methods for synchronization in internet applications to:– Reduce bandwidth requirements– Increasing latency tolerance

• Involves independent state streams that:– Are synchronized and predicted with different

methods according to their properties– Expose necessary model modifications

26

End of Communication TalkQuestions?

Multi-user Extensible Virtual Worlds

27

Combining Incremental and Parallel Methods for Large-scale Physics Simulation

Assets, Dynamics and Behavior Computation for Virtual Worlds and

Computer Games

28

Review of work to date


Computer Games

29

ScalableEngine

• Built to handle large VR environments efficiently (massive object count, low activity)

• Only physics system capable of handling Scalable City in real time

• Overhead proportional to level of activity rather than environment scale or object count

• Novel broad phase1 and physics pipeline2 methods published

1. Efficient Large-Scale Sweep and Prune Methods with AABB Insertion and Removal. IEEE-VR 2009

2. Accelerating Physics in Large, Continuous Virtual Environments. Concurrency and Computation: Practice and Experience, 24(2):125134, 2012

30

31

level of activity

Lower asymptotic complexity: order of magnitude performance improvement!

ScalableEngine:Broad Phase CD

32

Note: only a constant number of bodies undergoing active physics computation.

Excluding unnecessary work: Again lower asymptotic complexity

ScalableEngine:Full Physics Pipeline

ScalableEngine:Multi-user System

• Scalable City developed into massively multi-user client-server system.

• Player count increases activity level

• For multi-user, other factors matter as well

• Computational efficiency

• Parallelism

33

ScalableEngine

• Best engine at handling large environments

• Heavy computation similar to other software

• As activity increases, advantage matters less

• Regions with high activity see less benefit!

• Parallelized ScalableEngine by multi-threading all aspects of computation

• Improved performance, but not enough for massively-multi-player.

• Traditional physics does not parallelize well.

34

35

Limited parallelism in traditional physics methods

ScalableEngine:Multithreaded Physics

CLEngine

• Developed new physics simulation system from scratch focused on massive parallelism

• Based on work of Thomas Jakobsen1.

• Design modified for parallel application

• OpenCL utilized for portability to various compute devices (CPU, GPU, Accelerators)

36

CLEngine:Core

• Object representation broken down to particles and stick constraints

• Rigid body volume behavior is emergent

• All constraints independently solvable

• Very fine-grained, highly parallel core

37

CLEngine:Host Interface

• OpenCL weakness: expensive communication on dedicated GPUs

• Designed to reduce communication by

• Keeping many contiguous stages on the card

• Accelerate communication with transport kernel

• Reduce communication to state deltas

38

Contact GraphColl. Det. Integration

IntegrationColl. Det.

CLEngine:Performance

• 3-6 times CPU performance for single thread

• Higher parallelism acceleration curve

• Many optimizations still not done!

• GPU targetable for extreme performance

• Optimizations are more critical

• Communication, local memory, vector types

39

CLEngine Prototype Limitations

• Designed for an “all active” system

• Not state-aware, no incremental processing

• Does more total work than current CPU engine

• We want both advantages simultaneously!• Multiple ways to achieve this

• Challenges imposed by slow communication

• Integrating a broad phase solution efficiently

• Reporting results usefully and efficiently

40

Work Finished, Pt 1

• Made CLEngine & Testing Framework portable to OpenCL v1.1 systems generally• Tested on IBM/PowerPC, Ubuntu, Windows, OS X Intel

• Built high-level services on CLEngine core• Allows to interface like a traditional physics engine

• Ported Scalable City Server to VS2010 for OpenCL• Tools were 7 years old, OpenCL vendors didn’t support

• Integrated CLEngine as run-time option for S.C.• Not operational due to incomplete interfacing options

41

Broad Phase Integration:Where we’re going and why


Computer Games

42

CLEngine Broad Phase:Options Prev. Discussed

• Hash Grid

• Query stage for medium size (Lots, Cyclones)

• Multi-sort Sweep & Prune

• Single solution for small-medium

• Better performance for object clustering?

• Space-filling curves

• Reduce S&P sorts from two to one!

• All cases: Host must deal with large objects

43

Work Finished, Pt 2

• Space-filling curves with S&P implemented

• Morton: massive false positives

• Hilbert: high false positives & false negatives

• Space-filling curves generally too inaccurate

• Multi-sort S&P has similar limitations to Grid

• Parallel last pass inefficient w/o similar object size• Ideally precisely same object size for symmetry

• Traversal must stop based on largest object size

• Load balancing also affected by clustering

44

CLEngine Option 2:Broad Phase on Host

• More flexible, generally performant

• Handles all object sizes well

• Thread parallel, incremental processing

• CLEngine sees only relevant object subset

• Active objects & objects overlapping them in B.P.

• Maintained by communicating deltas

• CLEngine core is simpler: process & report all

• Focus optimization on communication

45

Work Finished, Pt 3:Host Broad Phase Design

• Designed Host Broad Phase System

• Communication manager being implemented

• BP and state system used to consolidate delta

• BP optimized for thread-parallel high activity

• Doubled performance under these conditions

• Free-threaded interfaces for all operations

46

End of Physics TalkQuestions?


Computer Games

47

Diagnosing Unbounded Heap Growth in C++

• Project Description:

Improper memory management in software can cause performance to decline and eventual program failure. These problems can be difficult to detect during testing due to the unpredictable amount of time it can take to exhibit overt symptoms, and those symptoms may appear unrelated to memory management. The purpose of this research project is to identify causes of unbounded heap growth in C++ software beyond traditional memory leaks.

• Major Accomplishments:– Heuristic perfected to yield low false positives/negatives with continuously

improving accuracy over time– Identified memory problems in Google Chrome, WebKit, Ogre3D– Fixed growing data structures in Chrome and Ogre3D

48

Review from last meeting


49

Diagnosing Unbounded Heap Growth in C++Motivation

• Scalable City needs to run continuously– Many months without intervention/access– Had slow growth of memory

• leading to crash after several weeks

– Available analysis tools reported no leaks!

• Software frees all memory correctly!– Different kind of undetected memory issue

50

Diagnosing Unbounded Heap Growth in C++Problem Description

• Types of Unbounded Heap Growth– Reference Lost (Leak)

• Reference lost to memory without freeing it• Well studied, wide variety of tools that identify leaks

– Reference Retained• But no longer needed• If growing, it’s accumulating in a data structure• Few tools exist to deal with this problem in C++

– Valgrind, IBM Rational Purify, Insure++ do not detect this• Generally also referred to as a leak

51

Diagnosing Unbounded Heap Growth in C++Problem Description

• Memory Tumor– Data structure with unbounded growth– Examples:

• Software cache that is never flushed• Aggregates that are improperly maintained

52

Diagnosing Unbounded Heap Growth in C++ State of the Art

• Valgrind: tracks allocation calls• Staleness Detection (SWAT, Hound)

– identifies memory that hasn’t been accessed for a long time

– Misses many potential tumors by design• Unneeded memory can still be accessed (iterated over)• A brand new allocation can contain regions with “stale” objects

– std::vector<Object>::resize()

• LeakBot (IBM)– Relies on Java reference graph– Not focused on unbounded growth

53

erik

new slide

Diagnosing Unbounded Heap Growth in C++ State of the Art

• Theory behind our test– Unbounded heap growth occurs in aggregates– Healthy aggregates have two stages

• Its maximize size grows during execution• Its maximize size has stabilized

– Our test detects aggregates that grow without stabilizing

54

erik

new slide

Diagnosing Unbounded Heap Growth in C++Operation: User Perspective

• Textual global search-and-replace– Replace STL & boost library headers with ours– Install our namespace over aggregate types

• Create automated test for your application– Choose test point to perform sampling

• Run application for hours or days– Test becomes more and more accurate over time

• Turn system on or off as compile option

55

Diagnosing Unbounded Heap Growth in C++Operation: Internal Perspective

• C++ has no reflection (low-level language)– Advanced C++ features utilized to overcome

• Code injection: over-ride aggregate classes– Central structure tracks all aggregate instances– Constructors & Destructors maintain structure

• Sampling: Periodic size queries• Identification heuristics

– Use size history to note candidate tumors

56

Diagnosing Unbounded Heap Growth in C++Tradeoffs

• Trade-offs– Source Code tumor aggregate identification

• Cannot over-ride constructors in automated way• Identified by: full type & instantiation order

– Emphasize machine time over engineer time• Cost of continuous test is low compared to inspection

– Find all unbounded growth• Don’t focus on specific cause: keep it simple

57

Additional Work


58

Diagnosing Unbounded Heap Growth in C++Progress

• Language Compatibility Improvements– Dependence on RTTI eliminated

• __FUNCSIG__ and __PRETTY_FUNCTION__

– Forward declaration of template arguments• Compatibility with non-standard C++ practices• C++11 solution or boost alternative

59


• Low memory overhead– Data set used to grow over time during test

• Limited length of test in practice

– Now store constant overhead per aggregate• Results equivalent to previous system

– Use exponentially-growing test interval• Can deploy software with active tracking

– Use real user input vs automated testing

60


• Multithreaded applications supported– Opens testing to larger class of programs

• Multi-threaded sampling acceleration designed– Not yet implemented– Lowers time overhead– Good for active deployment option

61


• Stack tracing option for aggregate identification– Higher run-time overhead– Full identification resolution

• Default system identifies aggregates by:– Full type of aggregate– Instantiation order

• May not be enough in some circumstances– Non-deterministic instantiation order &– High number of instances of specific types

62

Diagnosing Unbounded Heap Growth in C++ Results

• Overhead– No Call Stack Tracing

• Minimal effect on execution time and virtual memory usage• Software can be deployed with Growth Tracking enabled

– Stack Tracing• Significantly slower and increased memory usage

63

erik

new slide


• Testing performed with 3rd-party software– Expanded testing to identify problems– Influenced many of our improvements– Tests efficacy & applicability more extensively

64

Diagnosing Unbounded Heap Growth in C++Results

• Ogre3D Rendering Engine– Identified 2 tumors– GPU memory allocation (instancing module)

• Fix integrated into Ogre 1.7.4

– Improper map maintenance for multiple scene manager support

• Bullet Physics Engine– Core confirmed clean– 1 tumor in demo framework (not in physics core)

65

erik

new slide


• Testing Against Third Party Applications– Complex applications outside our experience– Most well-tested, competitive software– Web Browsers: Google Chrome at top of list

• Valgrind• Custom in-house heap analysis tools

– Compare Safari (WebKit), Firefox also– Navigated to 1000 popular websites in a loop

• Challenge: complex, custom build environments– Not an applicability issue in-house

66

erik

new slide


• Google Chrome / Chromium– GrowthTracker identified 21 tumors– Fixed most serious tumor ourselves

• Integrated into Chrome 18.0.1025.142

– Exposed limitation: sub-aggregate relationships

• WebKit (Safari Browser)– From preliminary partial conversion– Identified 2 tumors

67

erik

new slide

End of Heap Memory TalkQuestions?

Diagnosing Unbounded Heap Growth in C++Questions

68

Documents

Bandwidth Reduction and Latency Tolerance in Real Time Internet Applications Multi-user Extensible Virtual Worlds 1