evans hagedorn GDC07 public

Notes on downloaded slides: The accompanying videos can be

downloaded from: www.ChrisEvans3D.com/tutorials.htm

The videos were captured with various software which sometimes seriously butchers the framerate, please keep this in mind

We have added comments generally summing up what we said about each slide

The audio of our session in mp3 format can be purchased from ‘GDC Radio’ (http://store.cmpgame.com/product.php?id=1847&cat=)

http://store.cmpgame.com/product.php?id=1847&cat=

http://store.cmpgame.com/product.php?id=1847&cat=

Christopher EvansSenior Technical Artist, Crytek

Responsibilities: R&D Artist Artist tools Getting characters in-game

Hanno HagedornSenior Character Artist, Crytek

Responsibilities Character Heads

Modeling Texturing Rigging

Facial Setup Character Bodies (partially)

What is this stuff? A cross section of current

character-related R&D at Crytek

Mostly WIP, all real-time, real assets

Presenting collaborative work Artist-driven, or real world

data-driven tools

Intended Audience Intermediate to Advanced –

Requires experience and familiarity with the subject

Technical Artists (TA) and artists with technical interests

Not too complicated Simple, pragmatic approaches

Session Overview Low Level

Deformation How you drive deformations How you deform geometry

Animation High Level

Character Variation Case Studies Internal Tools Asset Creation Facial Editor

Setup Animation

What is a Character?


Person or creature?

Rejected internal concept art


Vehicle or machine withmoving parts?

Rejected internal concept art


Any object or set of objects whose movement is derived from a skeleton.

What is a Character? People and creatures Vehicles and machines Moving character attachments

Guns, backpacks, heads, hands Dynamic, jointed physics

simulations Ropes, complex constraint setups

Pre-baked physics and broken objects Pre-baking cinematic destruction

VIDEO: VIDEO: 00:0400:04

The Low Level:

Deformation

George Bridgman, The Human Machine 1929

The Low Level:

Deformation What drives the

deformation? Motion? Arbitrary inputs? Shader-based?

How do you move and deform geometry? Is it efficient? What kind of approach?

VIDEO: 00:45


The Low Level:


deformation? Motion? Arbitrary inputs? Shader-based? (procedural noise)

How do you move and deform geometry? Is it efficient? What kind of approach?

Driving Deformation:

Pose Driven Systems


Driving deformation:

Pose-Driven Systems

You can drive many things with skeletal poses: Morph Targets Joints Maps

Outside our industry: Not a new Not a new idea at allidea at all

Simple 3 Vector Driver: Shape


Simple Driver: Color and Shape



Pose-Driven Systems

Pre-baked Baked to keys/shapes

Runtime In engine

Pros/Cons:

Pre-Baked Pose-Drivers Pros:

Setup is in the asset creation app Implementations work well in most

animation-based games (joints) Cons:

Non-dynamic - Does not work with motion generated at runtime

Ragdoll physics Physically-based motion Other procedurally-generated motions


Pose-Driven Systems

Pre-baked Baked to keys/shapes

Runtime In engine


Runtime Pose-Drivers Pros:

Works dynamically in any situation Only limit is the number of poses you

define Cons:

Difficult to set up Need to mirror in the asset package

Constraints Parameterization


Arbitrary-Drivers Holy grail of driven systems General Arbitrary Drivers

General input/output system Houdini, Shadetree, etc…

The more general the better Runtime expressions?

Not limited to deformation at all!Not limited to deformation at all!





The Low Level:


deformation? Character motion? Skeletal motion? Arbitrary inputs? Shader-based? (procedural noise)

How do you move and deform geometry? Is it efficient? Is it an Artist-centric approach?

Deformation:

Skinning


Linear Skinning Weight vertices in a linear fashion to n bones

Definitely nothing new First use in a game (software)

Trespasser? (1997-1998) First use in a game (hardware)

dx8 via vert shaders dx7 rarity

Hardware ‘deceleration’

Linear Skinning,or smooth skinning, or linear blend skinning..

Pros Industry Standard Lots of available research

Cons Poor volume preservation

The more a joint bends, the more it collapses Difficult to get specific control Can be more resource intensive

“Hardware Skinning with Quaternions” (2004) by Jim Hejl (EA) Game Programming

Gems 4 First game to utilize it?

Our Implementation of:

Spherical Skinning

Pros/Cons

Spherical Skinning

Pros Much better volume preservation Less resource usage

(Skinning position, tangent, and bi-normal)

72 bones per draw call ~48 (Linear w/ matrices) ~72 (Linear w/ quats)

20 instructions ~40 instructions (Linear w/ matrices) ~60 instructions (Linear w/ quats)



Cons Needs support in the asset creation

package Plugin, preview, etc…

Inherent issues with some weighting configurations

N bones per vert Non-consecutive or interleaved bones

Pros/Cons

Spherical Skinning

Interleaved BonesExample: Character Studio Biped

Cons Needs support in the asset creation

package Plugin, preview, etc…

Inherent issues with some weighting configurations

N bones per vert Non-consecutive or interleaved bones

The volume preservation sometimes corrects things we have come to rely on linear skinning for

Pros/Cons

Spherical Skinning

Interested in New Skinning Technology?

GDC Session: Skinning with DualQuaternions

Tomorrow (Fri)9am – 10amRoom 2009, West Hall

Deformation:

Morph Targets


Deformation:

Morphs are Your Friends Also known as:

Blendshapes, shapes, vertex animation Artistic approach – Artist

defines the look completely Facial animation

Sculpt muscles, phonemes, visimes Corrective shapes

Driven by arbitrary inputs Baked simulations

Bending metal, ocean waves, soft tissue

Keep them light in engine Sparse morphs

Only the verts that move Corrective shapes can be *very* light Only move the points that you have to

Sparse morphs on DX10 hardware Sparse morphs running on dx10 hardware

Supports binding to more than 4 color images Store morphs in a texture Blend between textures

There is a demo in the DX10 SDK They also have wrinkle maps in this example!

Deformation:

Morphs are Your Friends

Driving Morphs

Pose-Driven morphs Artist-Sculpted Pipeline

Define pose vectors Artist sculpts the pose Link sculpts to those pose vectors Parameterize/blend between poses

Simulation-Sculpted Artist creates deformation Links deformation to some arbitrary

parameter in the engine

Our asset creation tool Allows the artist to pose the rigged

character We decided to use pre-defined pose vectors

Generates a mesh in that pose for him to sculpt

Extracts the artists sculpt with the skin deformation culled

Created a shape for that pose

Driving Morphs



Our In Engine Setup Apply the shape locally before the

skinning Assign the shape to the correct bone and

pose vector This is the simplest scenario

Single pose vector


Corrective morph shapesSAY GOODBYE TO SAY GOODBYE TO

SKINNING ARTIFACTS!SKINNING ARTIFACTS! Complex deformation

You don’t have to sculpt these You don’t have to sculpt these targets by hand!targets by hand!

The sky is the limit

This is a simple test, there are many more uses!



Deformation:

Helper Joints


Helper Joints Jason Parks: Advanced

Deformations on Runtime Characters – GDC 05/06

http://www.jason-parks.com/HelperJoints/ http://www.jason-parks.com/MuscleSystems/

Great but less artist-driven Universal Needs a lot of technical support

Non-intuitive Examples

Same driving issues as with morphs Pre-baked vs. Runtime

http://www.jason-parks.com/HelperJoints/

http://www.jason-parks.com/MuscleSystems/


Example: Shark

Example:Example:NanosuitNanosuit



Helper Joints (12):Helper Joints (12): Deltoids (shoulders) (8 joints)Deltoids (shoulders) (8 joints)

4 on each shoulder4 on each shoulder Pectoralis (chest) (2 joints)Pectoralis (chest) (2 joints)

1 per side1 per side Latissimus (back) (2 joints)Latissimus (back) (2 joints)

1 per side1 per side


All these systems, when do I use what? Dependent:

On your pipeline Result you want to achieve

The Low Level:

Animation


The Low Level: Animation IK Retargetting

Look IK Analytical IK for arms/legs

Feet on uneven surfaces CCD IK

7 DoF for bone chains Used for retargeting and planting

Parameterization + IK can yield good results IK fallback for dynamic motion


Character Variation

Character Variation

System supports 3 types of variation: Morph-based Attachment-based Decal-based

Character Variation Morph-based

Thinner/fatter body types Change size/position of design elements Characters divided up into ‘regions’ to

mask morph variations (vertex colors) Attachment-based

Skin Attachments Object Attachments

Decal-based Small 128px dirt, scars, camouflage, etc Layers on top of current texture

Super low resource usage

Case Study:

Variations for E3 2006

3 x 128px Camouflage Face Textures

1 x 128px Dirt Texture 2 Head Variations 2 Body Variations

Thinner/Fatter body types, Longer/Shorter clothing

Attachments Skin Attachments

Hands, Heads Object Attachments

Pouches, Helmets, Ammunition

E3 Character Variations 1/10










Issues We Ran Into

Morph-based Maintaining point index in LODs (variations)

Too Much Detail? Not enough? Designer setup time

Procedural runtime generation (examples-based)

How much of a difference can the player perceive ingame?

Parameterization Non-Face attachments

Eye placement Bone attachments

Case Study:

Character Flash Freezing Initial Freezing

Must be dynamic Any character in any pose or animation Slowly blend, do not ‘pop’ to a stop

Broken Pieces Physicalized parts

Interactive, physicalized parts Realistic

Actual parts

Case Study:

Character Flash Freezing Implementation:

Frozen Mesh Procedural ‘Ice Shader’ (layer) Brittle Physicalized mesh

Shattering Ice particles emitted from joint phys proxies Broken human bodyparts are created for most

of the bones in the character. Pieces are spawned in the orientation of their

associated joints Broken pieces

Left as individual physicalized entities Pickable


Baking Out Simulations: Isn’t That Cheating?

Baking Out Simulations: Isn’t That Cheating? Same thrilling game experience

for mid-low end users (though less interactive)

Benefits: Musculo-skeletal systems/Organic

Deformation Bake to joint-driven morphs Bake to joint-driven joints

Rigid-body physics Bake rigid body physics to keyframe data

1 joint per piece

Baked but still interactive?

Baking Out Simulations: Interactivity Organics:

Pose-driven joints or morphs Physicalization of Rigid Bodies

Use the keyframe data to generate accurrate impulse

Physicalize the animated rigid body upon collision

Physicalize after the animation plays Physicalize after the joint has reached x

distance from the object center. Physicalize when received damage

Pre-Baked Rigid Bodies


Baking Out Simulations: Characters at Heart Keyframe data

Parameterization Blend between multiple simulations Inherent issues, but for low spec: much

better than the alternative Compression Can have attachments

Deformible (bending metal etc) Can attach other characters to it

Baking Out Simulations: Issues We Ran Into

Draw calls Meshes need to be merged into one

Compression Piles of debris Export surroundings for

external sim Wavefront OBJ export

Terrain, objects, etc… Can run the simulation in Max, Maya,

etc… Better animator control for ‘Hollywood 11’


Hanno HagedornSenior Character Artist, Crytek

Responsibilities Character Heads

Modeling Texturing Rigging

Facial Setup Character Bodies (partially)

I am going to talk about...

Pipeline improvements Polybump2

How we do faces, from the texture to animation:

Asset Creation Textures Model

Shaders Facial Animation System

Crytek PolyBump2

Crunches assets with many millions of polygons

Capable of combining multiple models, each with many millions of polygons into one single calculation

32/64bit 3dsmax plugin or standalone application

Exports normal maps, displacement maps, and ambient occlusion maps

Gives out a storage file containing 100x the accuracy of a normal texture map

Extracting new data without recalculation

Unoccluded Area Direction

Simulates self shadowing inside the normal map while softening hard details

Combines ACC with the normalmap Softshadowing within the normalmap Applies a very soft look to the skin



Asset Creation

Get good texture reference Bad texture reference Synchronizing texture maps Get the right software

Get Good Texture Reference

We all love hand painted textures Hand painting * photorealistic / Schedule =

pain in the ass Well lit photos are key

Unwanted lighting will plague you later on The less lighting information will get

you closer to your desired result

Bad texture reference

Very colored Very blurry Shiny nose Lots of direct and

only few ambient light

Very contrasted shading

Good Texture reference

Equaly lit Very sharp No major highlights Interesting details Small

colorvariations in the skin

Synchronizing texture maps

Model and texture should feel like a single unit

Pore stamps can be very handy but… Never fit the existing texture Create a repetitive look

Make sure your diffuse and spec map complement your high frequency or pore-detailed model

Base Diffuse Texture

Normalmap

Specular Map

Diffuse with ACC lighting

Get the right Software

Z-Brush and Mudbox Great programs for sketching out faces and

shapes Great programs for polishing and detailing

assets Silo is looking very promising

Combining Modeling and sculpting into one workflow

Shader

Rim Lighting Sub-Surface Scattering (SSS) Eyes

Rim Lighting

Simulates greasy reflective Skin Works against the classic specular look of

last-gen games There is no such thing as ‘specular light’

in reality Similar to Fresnel reflections Adding ‘rim lighting’ by

Ambient color Cubemap Realtime automated cubemaps

No Rim Lighting

Basic Rim Lighting

Strong Rim Lighting

Sub-Surface Scattering (SSS)

Approximates light rays penetrating the skin and exiting another spot

Gives skin a nice reddish, fleshy look Even simulates thin vascular tissue

SSS-map allows us to define the skins thickness and color

No SSS

Basic SSS

Strong SSS

Eyes - Windows to the Soul

3dsMax screenshot


After several tests we decided to: Build a physically correct Eye model with

internal physiology Works better with new shader

possibilities Special Iris shader; glinting


3dsMax screenshot




Eyes - Self shadowing

3dsMax screenshot






Using overlaying geometry for emulating the shaded eye

Comments Eyelids and Eyeballs together This allows us to simulate the shadowing

of the eyelashes and surrounding bone structure

3dsMax screenshot


Self shadowing Using the same base as the Eyelids

saves textures and draw calls Eyeball moves under the Overlaying

Geometry while keeping the shading

3dsMax screenshot


Self shadowing Using the same base as the Eyelids

saves textures and draw calls Eyeball moves under the Overlaying

Geometry while keeping the shading

3dsMax screenshot



Eyes – In motion

Eyelids move along the Eyeball via Fleshy Eyelid morphs

Automated Saccadic eye motion Automated Eye Blinking, synced with the

Saccadic eye motion

Eyes – In motion

Eyelids move along the Eyeball via Fleshy Eyelid morphs

Automated Saccadic eye motion Automated Eye Blinking, synced with the

Saccadic eye motion

VIDEO:VIDEO:15:2315:23

Facial Animation System

Facial Animation System

Muscle Morphs Expressions

Linear Non-linear Phonemes Visimes

Joystick Controls Optical Flow

Morphs – The Base

Muscle Morphs 89 Morphs

Each muscle a separate morph A facial expression is sculpted by “muscle

morphs” Asymmetrical morph setup

Natural face movement Adds character to the facial movement

Automatic ‘fleshy eye’ movement Eyelids move synchronized with the Eyeball

Using same topology for all faces Easy transfer of morphs between meshes

Expressions – The Connection

Unlimited scalability Core system is very simple, yet very scalable

Multiple linear and nonlinear expressions

Can combine linear and non-linear Setting the base for the phoneme

extraction

Linear Expressions

Linear blending of morphs or drivers


Non-linear expressions

Non-linear blending of morphs Use multiple morphs to:

create a non-linear motion keep the volume of complex or twisted shapes


Expressions are drivers

System drives Morphs, Bones, and attachments All elements are treated equally Can be easily combined or parameterized

Phonemes Phonemes

The smallest distinctive units of speech, or the mouth shapes to make each sound

Strong and weak phonemes for appropriate usage

96 phonemes, 48 strong and 48 weak

Seamless blending between strong and weak phonemes

Phoneme Phonemes

The smallest distinctive units of speech, or the mouth shapes to make each sound

Strong and weak phonemes for appropriate usage

96 phonemes, 48 strong and 48 weak

Seamless blending between strong and weak phonemes

Visimes

Visual simplification of phonemes Phonemes like ‘o’, ‘u’, ‘oo’, and

‘ow’ are visually very similar 14 visimes, 7 strong, 7 weak Massively simplifies the work

with visimes Tweaking several phonemes at

once is handy when transferring the setup to a different character

Visual simplification of phonemes In use by 2D animations for over

90 years

Visimes

Muscle/Phoneme/Visime Breakdown

Phoneme Extraction Automated process

Automatically generates Phonemes from a pure Audio file

Manual process Breaks words into phonemes and places them

in the timeline corresponding to the audio waveform

barely needs touching up, easy to handle, batch-able.


Joystick Controls – Manual input device

User-friendly and intuitive Your Grandma could animate with this! Animator’s focus will be on animation and not

dragging sliders back and forth


User-friendly and intuitive Your Grandma could animate with this! Animator’s focus will be on animation and not

dragging sliders back and forth


Handles 4 diff expressions at once Example, Up/Down could be the

‘jawdropper’ and ‘jaw_presser’, Left/Right could be the ‘mouth_widener’ and ‘mouth_tightener’.

Automatically excludes opposing morphs

No more fighting with frown/smile or open/close

Puppeting Real-time Mouse input capture



Joystick Controls – scalability

User-definable Joysticks plug into the cure interface, any

expressions can be mapped to them Scalable toolset

Can be used to limit the detail an animator sees

You can predefine A, B, and C class joystick setups

We got inspired by Jason Osipa’s book: ‘Stop Staring’, which popularized joystick control setups

Optical-flow based Facial Motion Capture

Optical-flow based Facial Motion Capture

Getting good results with a minimum amount of work

No expensive hardware needed Animators here have used webcams at their

desks. Very easy to polish

MoCap system uses the same interface the animators do (joysticks)

Combines with the auto phoneme extraction to polish lip-sync

Theoretical Marker Setup

Practical Marker Setup

Marker corresponding joysticks

Marker corresponding joysticks


The Capturing Workflow


Final Result


Final Result

Casual B-Level animation: Used a Hand camera for Motion capturing Used Eyeliner for the Markers Untouched Facial Mocap Data Automatic Phoneme Extraction Procedural Eye blinking Procedural Eye behavior

Final Result


Upcoming Improvements:

Giving animations to the animation department for:

Enhancing and exagerating the acting Polishing up Mocap Data Cleaning up Mocap Data Keys

Improved Phoneme Blending Improvement will come, but our mainfocus is

on finishing up the game right now ;]

Questions?

Questions or comments?

Christopher Evans [email protected] [email protected]

Hanno [email protected]

[email protected]

For more info,come visit us at booth

XXXX

mailto:[email protected]




Documents

evans hagedorn GDC07 public