Upload
tristan-pralle
View
213
Download
0
Tags:
Embed Size (px)
Citation preview
Networking and Management Frameworks for Cluster-Based
GraphicsBenjamin Schaeffer
Integrated Systems Lab
University of Illinois
Syzygy
• Library for Virtual Reality on PC clusters
• Supports multiple programming styles– Distributed Scene Graph– Synchronized application instances
• Portable: Linux, Win32, Irix
• High performance and high quality
The Problem
• PC graphics and networking performance is becoming comparable to the graphics supercomputers popular in the 1990’s
• How can we exploit this?
• Unfortunately, PC’s can only host 1 high performance graphics card (AGP bus)
• PC cluster is the answer
The Problem
• PC cluster has much lower communications bandwidth between components (shared memory vs. network)
• This bottleneck means that PC cluster VR software must be highly optimized to get high performance
• Different applications will require different architectures. Difficult for the library writer!
The Problem
• Cluster applications are more difficult to manage than applications running on a single computer.
• Starting the application?
• Stopping the application?
• Configuring the application?
• What if the cluster is heterogeneous?
Related Work
• CAVElib: Cluster support in the latest version. Limited automatic data sharing. Sockets-based network facility for other data sharing.
• NetJuggler and Cluster Juggler: Two different extensions to VR Juggler with cluster support. Multiple copies of the application share an input event stream.
Related Work
• WireGL: Replacement for OpenGL shared library that broadcasts OpenGL primitives over the network to be displayed by cluster nodes.
• Avango: Shared scene graph for shared virtual environments. Can be adapted for cluster-based graphics.
Application Architecture
• Two basic application architectures (there can be more) for cluster-based graphics
• Client/server
• Master/Slave
Client/Server
• The application runs on a central node and sends rendering information to the render nodes. Uses a fixed protocol.
• Generic rendering clients run on the render nodes. These are the same for all applications.
• Examples: WireGL, Avango (?), Syzygy distributed database
Master/Slave
• Multiple copies of the application run, one on each render node.
• A master instance of the application controls what the slave instances do, via broadcasting state information each frame.
• No fixed protocol. Unique for each application.• Examples: CAVElib, Cluster Juggler, Syzygy
Master/Slave application framework
Pros and Cons
• Client/Server can require high bandwidth (WireGL).
• Client/Server can require software rewriting (scene graphs). These higher-level protocols can be network efficient.
• Master/slave in many cases requires low bandwidth.
• Master/slave might not be suitable for complex applications.
Syzygy
• Provides a common networking and management framework for building cluster VR applications.
• Why? Many different ways to do cluster VR, each appropriate for different situations.
• As practical examples, build a distributed scene graph and a master/slave application framework.
Communications Layer
Phleet I/O FrameworkMedia Protocols
Media Objects I/O Drivers
Application Frameworks
Messaging
• Syzygy communication is message-based.• Programmer can construct message types
and languages from those types.• Binary data format with built-in translation
between different machine architectures (big-endian to little-endian).
• Built-in message serialization and de-serialization
Communication Objects
• Syzygy includes server objects that can manage multiple full-duplex network connections. Communication occurs via Syzygy messages.
• Also client objects that communicate with the servers.
• Fault tolerance is built in. Objects can activate in any order, stop, restart, reconnect, disconnect, etc.
Synchronization
• BarrierServer object controls synchronization across the cluster.
• BarrierClient objects connect to it.• Built on the standard Syzygy client/server objects.• Synchronization groups are dynamic. Clients can
enter and leave the group at any time.• Fast. Buffer swap latency is comparable to the
ping time on the cluster network for ethernet.• Important for cluster graphics!
Synchronized Buffer Transfers
• For efficiency in graphics, information should be transferred once per frame. Small messages need to be buffered.
• Syzygy includes support for objects that automatically buffer messages in the background, transferring the buffers on the graphics buffer swap.
• Double-buffering of messages is used to get the most out of slow networks. While it is buffering the new messages, Syzygy is transferring the last buffer.
ser v
ercl
ient
clie
ntSend frame (n)
Fill frame (n+1)
Draw frame (n-1)
Receive frame (n)
Send Connection
Request
Send State
RecvState
Draw frame (n)
Idle Recv frame (n+1)
Idle
Send frame (n+1)
Fill frame (n+2)
Send frame (n+2)
Fill frame (n+3)
Draw frame (n+1)
Recv frame (n+2)
Draw frame (n+1)
Recv frame (n+2)
Phleet
• Distributed operating system (minimal)
• szgserver controls.
• szgd provides remote execution services.
• szgserver is merely a connection broker. Most communication occurs directly between programs.
Phleet
• Parameter database stored in szgserver. Meta-config file that can be managed from the command line of any computer.
• Global locks
• Simple message API routes through szgserver to all managed programs (like Unix signals). Can tell a program to reload its parameters or exit.
Distributed Scene Graph
• Client/server. One node serves geometry to the render nodes.
• Bandwidth is automatically conserved. Only scene changes are sent from one frame to the next.
• Semantic nature of scene graph helps ease programming.• Robust. Render clients can disconnect and reconnect
while the application is running. Dynamically change your display. Similarly, stop your application and start a new one without bringing down the render clients.
Distributed Scene Graph
• Built on a general hierarchical distributed database.
• Nodes in the database are created and later altered via sending Syzygy messages to the database.
• Incremental changes can be made to a node (for instance, altering just some points in a point set). Saves bandwidth.
• An API is provided to aid the programmer in generating these messages.
Distributed Scene Graph
• Node types include:– Transform – Points– Lines– Triangles– Texture coordinates– Textures– Visibility– Text Billboard– Etc.
SyzygyServer
Barrier Server
Graphics Server
Input Client
szgd
szgd
szgd
InputServer
Barrier Client
Render
Master/Slave Framework
• Master/slave = same application runs synchronized on render nodes.
• One node is the master and distributes I/O or other control info.
• Framework makes it easy to write a synchronized application, handle input devices, and manage the whole thing.
• A good way to write a custom distributed graphics protocol!
Master/Slave Framework
• Programmer registers memory chunks.• Event loop:
– Pre-exchange callback (computations), input device polling on master.
– Data exchange (handled by library)– Post-exchange callback. Input device state is
synchronized on all application instances. As is registered memory.
– Draw callback.– Synchronization.
Master Slave Slave
Barrier Server Barrier Client Barrier Client
Input Client
InputServer
Master/SlaveApplication
Detail
Practical Experiences
• Volume visualization using master/slave framework.
• Visualization of MPI-based codes using distributed scene graph.
• Quake 3 level viewer. Master/Slave.• Mathematical visualizations: Sphere eversion and
hyperbolic geometry. Master/slave.• Fully articulated avatars animated by motion
capture data. Distributed scene graph.