Geometric Algorithms in 3D Real-Time Rendering and

facial Expression Modeling

Prof. Marina L. Gavrilova

University of Calgary

Quote:"Perseverance is not a long race;

it is many short races one after another."-Walter Elliott

• Brief Overview of the Area

• Geometric Approaches for mesh reconstruction and image morphing

• Real-Time Optimally Adapting Mesh Algorithms

• Non Photo-realistic Facial Expression Modeling and Morphing Algorithms

• Swarm Real-time Visualization System

Presentation Outline

Objective:To showcase the power of the geometric and

topology-based approaches in 3D visualization and modeling

Relevant Areas

• Computer-Aided Design• Geographical Information Systems• Biometrics• 3D Animation• Computer Games• Strategic Planning• Defense• Other areas

State of the Art

• TERRAIN MODELING

• Multi-resolution dynamic mesh• Viewer-dependent refinement• Culling techniques• View frustum algorithms• Triangle-count preservation• Subdivision refinement• Controlled LOD

• Visual continuity and smooth transition are crucial

• Lindstrom, Hoppe, Duchaineauy, Hesse, Silva, Skala, Kolingerova

• EXPRESSION MODELING

• Multi-resolution adaptable mesh• Light source and view point

dependent rendering• Control points and areas• Tracking algorithms for transforming • Fitting algorithms for modeling• Wavelets to extract specific

characteristics• Visual continuity and feature

preservation are crucial

• Jain, Salesin, Gooch, Gordon, Thalmann, Plemenos, Sourin

Our Approach TERRAIN MODELING

Challenge: massive amount of data, smooth rendering

Methodology:• Multi-resolution dynamic triangle

mesh with new refinement strategy• New split and merge queue operations• Frame-to-frame coherence algorithm

for local updates• New geo-morph smoothing algorithm Outcome:• Visual continuity, rendering

efficiency, frame-rate constancy for massive terrain

EXPRESSION MODELING

Challenge: massive amount of detail, tracking changes

Methodology:• NPR approach to feature extraction and

modeling• New Distance-Transform algorithm for

morphing• Delaunay triangulation for mesh

representation• Image segmentation and feature-line

extraction algorithm

Outcome: Expressive detail preservation, efficient

storage, fast processing, 3D controllable deformations

Our Focus

• Topology preservation and geometry plays an important role in many algorithms, allowing for rendering efficiency, visual continuity and feature preservation.

Part I:3D Real-Time Massive Terrain Visualization

Sourcing data from real world

Understanding Height FieldsDEM: Digital Elevation Model

• Contains only relative Height

• Regular interval

• Pixel color determine height

•Discrete resolution

Converting Height field data into 3D topological mesh

200 255 150 100

100 255 255 200

200 150 200 100

• Pixel value (z) is used as Height Map

• Vertices are generated as points in 3D

• A Mesh is triangulated

Terminology

DEM: Digital Elevation ModelVDR: Viewer Dependent RefinementROAM: Realtime Optimally Adapting MeshCLOD: Continuous Level of Detail

What is mesh?

• It is often required to convert geometric objects in any other form to polygonal meshes of geometric simplexes (i.e. triangles) before rendering

• Meshes are used for computer modeling, biometric, GIS, cartographic, and animation applications

• Mesh optimization techniques are used to reduce the mesh to an acceptable size (user defined), minimizing the loss of quality (subject to some form of error analysis).

Mesh definition

• A mesh is defined as a pair (V,K) where V is the set of vertices and K is a simplicial complex specifying the connectivity of the mesh simplices (the adjacency of the vertices, edges and faces).

• The 0-simplices are called vertices, the 1-simpices are called edges, and the 2-simplices are called faces.

• Triangulation is an example of mesh.

Example: 3D Terrain Rendering

• Uses DEM as input of the application

•Generates dynamic mesh to achieve frame coherent animated view in real-time

Contribution• In our work we used the ROAM definition introduced in

[Duchaineauy], and developed a number of algorithms that rely heavily on topological properties of the modeled terrain in order to handle massive data sets.

• The observation of hardware performance reacting to strip length allowed us to design a fast and effective greedy approach to dynamically maintain triangle stripping. The method resulted in achieving rendering efficiency and frame rate constancy.

• We further improved the rendering efficiency while preserving important terrain features utilizing topological information.

• We eliminated the priority queue for merge operation, amortized update cost by allowing split queue to degenerate, took advantage of frame-to-frame coherence for local update and devised an efficient method for mesh-smoothing using geo-morph while preserving important terrain features.

Methodology applications

• Massive terrain rendering

• Realistic deformation modeling using mass spring system

• Adaptive spatial memory subdivision in robot motion planning simulation

Basic Approach

• The entire optimization process can be viewed as an optimal spatial subdivision problem. There exist only two legal moves to achieve optimality: subdivide and merge.

• Models that are based on this concept do not require computationally extensive optimization process. Instead, the mesh progressively adapts to an optimal solution.

• These models usually define a multi-criteria based error metric to assess the energy of the mesh.

Conventional Approach

• Regular Grid and TIN: Triangulated Irregular Network• Problems:

– Significant preprocessing– Generates too many triangles– the visible area might constitute a tiny fraction of that detail– Viewpoint dependent refinement is not possible with static

meshes. – Have a bad triangle to screen pixel ration.– Unable to obtain a steady performance.

• Unable to Render Massive Terrain

Problems with static mesh

d

h h

21 unit

a2

a2h

h1

1d

d

2p

2a p

2 tan2

h da

1screen pixel ratio 1:r h h

Problems with static mesh …

Detail

98.5%

1.5%

Screen

10%

90%

A Static Mesh (40,000 tri)

Dynamic mesh

• Dynamic mesh generation is based on real-time adaptive models that allow local transformation of a given mesh to different levels of details and adapt progressively to the changes in the rendering environment.

• Dynamic models use some form of hierarchical data structure to store a mesh in multiple LOD.

• There are many different versions of dynamic mesh currently under research and development. Several patented technologies have been released in the industry (i.e. Intels MRM). The two most prominent approaches are Progressive Meshes (PM) and Real-time Optimally Adapting Mesh (ROAM).

ROAM :•Can Subdivide when more details necessary

•Merge & Split Queue

•Tree Structured

ROAM (25,000 Triangles)

The ROAM Advantage

ROAM advantages

• An efficient top-down dynamic mesh model. • Display of complex surfaces can be provided by

dynamically computing a multi-resolution triangular mesh for each view.

• One can minimize geometric distortions on the screen while maintaining a fixed triangle count.

• Visual discontinuities could be minimized, and efficient mesh corrections can ensure selected lines of site, or object proximity are correctly represented.

• An incremental priority-queue based approach in ROAM allows exploiting frame-to-frame coherence for fast efficient computation of the adapting mesh.

Geometric subdivision

Problems with Geometric Subdivisions

ROAM principle

The basic operating principle of ROAM

Quadtree and Bintree for ROAM

Comparison between Quadtree (top) and Bintree (bottom)

ROAM Model

DEM

Inte

rpola

tion

En

gin

e Triangle Bin Tree Data structure

Render

Ref

ine

Err

orM

etri

c

Speed: 15.0 FPS

Size: 15000 Tri

Our ROAM Model

Speed: 45.0 FPS

Size: 19000 Tri

DEM

Inte

rpola

tion

En

gin

e Triangle Quad Tree Data structure

Err

orM

etri

c

Ref

ine

Render

G Q

Wavelet

ErrorAnalysis C

lean

up

Op

era

tion

ROAM Implementation and Improvements

• Pre-process DEM• Reduce overhead for maintaining Split queue• Use degenerated split queue• Replace priority merge queue (HEAP) with a

simple FIFO queue.• Merge arbitrary obscured regions (outside visible

frustum) only • Improve smoothness, reduce popping effect

Proposed Unique Geometry-based methodology

• DEM Analysis

• Adaptive Loop Subdivision

• Geo-Morphing

• Visible frustum computation

• CCM - Cascading Coupling March method for massive terrain

Preprocessing of DEM input

• Convolute the surface (2nd Difference)

• Efficient LOD distribution

• Used in Advanced Shading (feature preservation)

Differentiating the Surface: Convolution

Original DEM of Kluane Park First differentiation. Blue area Shows steep mountains.

Final LOD distribution image Shows rocky areas require more details

Second Differentiation. Blue areas indicate steep mountain peaks.

Differentiating the Surface (cont’d)…

• Take Weighted Average of neighboring pixel Gradients

• 2 passes gives us the rate of gradient changes

Statistical Analysis of ROAM

• Bottleneck: Split and Merge Operation and visibility test• Only 2.7% triangles are merged within the visible frustum• 75% of split operations occurs on closest 5% region of the

visible frustum• Taking above into account, special priority function for

split queue was developed based on U=Uniformity • F=Frustum priority• D=Curvature priority• P=Distance priority• DET=Curvature (preprocessed)• Lev(f)=Subdivision level of face f

Methodology: Speeding up the process - Amortizing refinement

• Updating the Split queue over multiple frames

• Allow split queue (heap) to degenerate• Refine integrity using error threshold test• Re-generate queue after constant number of

frames• Dramatic gain in speed

Methodology: Maintaining Degenerated S-Queue

Methodology: Geo-Morph

• Smooth interpolation of vertex over subdivision

• Requires maintenance of one additional queue (G-queue)

Methodology: Smoothing

• Compute face normal • Compute vertex normal• Local computation of normal

during refinement (Geo-Morph)

• Apply vertex normal when rendering

• Advanced shading: Linear combination of face and vertex normal (Hybrid)

Example of smoothing

Result with Texturing

Massive Terrain Model: ALS

Adaptive Loop Subdivision and Cascading Coupling March

Quote:“Map of a region was a size of a city, map of an

Empire was a size of a Province."Jorge Luis Borges

Loop Scheme

Patching alignment

A

B CA

BCFaces:

•Regular

•Tface

Regular Tface

Adaptive Loop Subdivision

Recursive Split

A

A

A

A

BB

B

B

C C

C

C

AB CA

BC

Split

•Prefer: Tface

•Retract Tface

Patch Stitching: CCM

“CASCADING COUPLING MARCH”

Method Description

• We develop CCM to stitch two quadrants of the adaptive mesh patch, and to ensure seamless connectivity by progressively refining and connecting the subdivision tree (quadtree) data structure.

• We ensure integrity of the data structure among multiple patches so that subdivision and simplification are not limited to the boundaries of the patches.

• CCM allows to consider the entire data structure as a number of strongly connected sets of quad-trees. As a result we can refine the mesh as one seamless continuous structure (not patches stitched together).

CCM idea

• We perform a coupled recursive march of the two quad-trees from the top to the bottom of the tree, subdividing or connecting border triangles.

• To increase geometric regularity we used non-orthogonal axis and used space warps to convert the scheme back to orthogonal axis for rendering. This is crucial when we are performing viewer dependent refinements. Patch locating and coordinate transform algorithms are designed to maintain a list of visible patches. Additional patches are added and connected using CCM as the simulation advances. Obscured patches are deleted using recursive march.

• This new method allows a greater flexibility, efficiency and control over smoothness

Two Adjacent Patches

Snapshot 3 … Level 2

Results on Mesh Rendering

- Strategies evolved, Runtime and other aspects

Results (Output obtained)

(a) 0, 1, 4curv dist ur r r= = = (b) 0, 1, 6curv dist ur r r= = = (c) 0, 1, 8curv dist ur r r= = =

(d) 1, 0, 4curv dist ur r r= = = (e) 1, 0, 6curv dist ur r r= = = (f) 1, 0, 8curv dist ur r r= = =

Results (Runtime) (Cont’d)Frame Per Second - Modified ROAM Scheme

0

20

40

60

80

0 10 20 30 40 50 60Time (sec)

Fra

me/

Sec

ond

Kluane 25000 Alaska 35000Fractal 55000 Kluane AveAlaska Ave Fractal AveKluane Alaska Fractal

55.15 39.12 27.87

Average Frame/ Second

Results (Runtime) (Cont’d)

Rendering Statistics for Modified ROAM Scheme

0

5000

10000

15000

20000

25000

0 50 100 150 200

Time of Simulation (Seconds)

Num

ber

o T

rian

gles

Total T riangles Visible T riangles

Results (Runtime) (Cont’d)Average Load of The Geomorph Queue

in ALS Based Models

0

500

1000

1500

2000

2500

3000

3500

Number of Triangles

Aver

age

num

ber

of

ver

tice

s

in t

he

Geo

morp

h q

ueu

e

20 frames 286 427 567 717 805

60 frames 791 1097 1446 1781 2012

100 frames 1019 1746 2382 2842 3276

20000 25000 30000 35000 40000

Summary of Results

• High Speed (45 FPS average) for 20000 faces!!• Static mesh upper limit: 55 FPS• Continuity is achieved (Geo-Morph) – 300

vertices on average for 45 FPS –very small overhead

• Visual appeal (smoothing & Texture)• Less variant Frame rate• Efficient Understructure (amortized update)

More Results on Mesh Rendering

See Animation Demos