Introduction to Data-driven Animation

Introduction to Data-driven Animation

Jinxiang Chai Computer Science and Engineering

Texas A&M University

Data-driven animation

- Motion graphs

- Motion interpolations

- Statistical motion synthesis

Outline

Motion capture data are easy to capture

But we cannot capture all kinds of motion variations

- different subjects - different styles - different emotions

Key idea: reuse prerecorded motion data to achieve new goals!

Motivations for Data-driven Approaches

Data-driven Animation

Goal: convert motion data into a usable form.

Can we automate this?– Must preserve realism and provide control

Motion model User specifications

MotionMotion

processing

Motion data

Data-driven animation

- Motion graphs

- Motion interpolations

- Statistical motion synthesis

Outline

Motion Graphs: Key Ideas

Given: lots of prerecorded motion clips

Concatenate them to create new motions!

Motion Graphs: Key Ideas

Given: lots of prerecorded motion clips

Concatenate them to create new motions!

Maze

Motion ConcatenationMotion capture region Virtual environment

Obstacles

Sketched path

Motion ConcatenationMotion capture region Virtual environment

Unstructured Input Data

A number of motion clips • Each clip contains many frames• Each frame represents a pose

Unstructured Input Data

Connecting transition • Between similar frames

Graph Construction

Building Motion Graphs

So how can we find transition points between motion clips?

Building Motion Graphs

- Every pair of frames has a distance.

- Transitions are local minima below a threshold.

Motion 2 Frames

Mot

ion

1 Fr

ames

Finding Similar Frames

• Need derivatives (velocity, acceleration, etc.)

• Compare motion in joint angle space or 3d point space?

• Must account for coordinate invariance

– Different camera ≠ different motion!

Distance Metric

For more detail, refer to [Kovar and Gleicher, Lee et al]

Finding Transition Points

Transition thresholds control quality vs. flexibility tradeoff.

Threshold = 0 cm Threshold = 8 cm Threshold = 16 cm

Structures of Motion Graphs

Motion data structure: a graph of frames/poses

Avoid dead-ends: finding strongly connected components

Contact states: avoid transition to dissimilar contact state

Interacting with Motion Graphs

So given a motion graph, how can we generate an animation sequence?

Interacting with Motion Graphs

So given a motion graph, how can we generate an animation sequence?

- Random graph walk: Any sequence of edges is a motion!

Using Motion Graphs

How can we control synthesized motions (e.g., moving from point A to point B, speeds, walking directions)?

- Graph search: Find graph walks that minimize a cost function.

Path Synthesis

Goal: extract motion that follows a path.

User’s path ( )

Motion’s path ( )

Minimize

2)()( i

ii sPsP

P

P

Motion Control

Goal: extract motion (M) that satisfied constraints (C) specified by the user

Minimize

),( CMG

Results

See videos [click here]!

Discussion

Pros: + Fully automatic: work on unstructured data + High-quality animation: motion concatenations + Easy to control: graph search

Cons - Poor generalization: cannot produce new poses - Control accuracy: cannot generalize new poses - Not compact: needs to retain original mocap data - Scalability

Data-driven animation

- Motion graphs

- Motion interpolations

- Statistical motion synthesis

Outline

Motion Interpolations: Key Ideas

Given: lots of prerecorded motion clips

Interpolating motions to achieve new goals!

Motion Interpolations: Key Ideas

In research: more than decades [e.g., Rose et al. 98]In games for a long time

- Interpolating motions needs build correspondences between motion examples- Thus, motion interpolations require structurally similar motion examples!

Canonical timelinet

Time warping functions

w(t)

t

Motion Decomposition

Canonical timelinet

Time warping functions

w(t)

t

Reference motion

Motion Decomposition

Canonical timelinet

Time warping functions

w(t)

t

Contact Transitions

Motion Decomposition

Canonical timelinet

Time warping functions

w(t)

Motion 1

t

Motion Decomposition

Canonical timelinet

Time warping functions

w(t)

t

Motion Decomposition

Canonical timelinet

Time warping functions

w(t)

t

Motion Decomposition

Canonical timelinet

Time warping functions

w(t)

t

Motion Decomposition

Using dynamic time warping!

Canonical timelinet

Time warping functions

w(t)

t

Motion Decomposition

Canonical timelinet

Time warping functions

w(t)

Motion 2

t

Motion Decomposition

Canonical timelinet

Time warping functions

w(t)

t

Motion Decomposition

Canonical timelinet

Time warping functions

w(t)

t

Motion Decomposition

Canonical timelinet

Time warping functions

w(t)

t

Motion Decomposition

Canonical timelinet

Time warping functions

w(t)

t

Motion Decomposition

Canonical timelinet

Time warping functions

w(t)

t

Motion Decomposition

Motion Representation

Registered motions Time warping functions

Contact Transitions

Motion Annotation

Preprocessed motions mi

Motion Annotation

Preprocessed motions mi

For each motion mi, we annotate the motion with control parameters si

- such as walking speed, direction, step size, kicking directions and positions, etc.

Motion Annotation

Preprocessed motions mi

Motion space: m

Control parameter space: s

Motion Interpolations and Control

Preprocessed motions mi

Motion space: m

Control parameter space: s

How can we generate an animation that achieves the goals specified c* by the user?

Motion Interpolations and Control

Preprocessed motions mi

Motion space: m

Control parameter space: s

How can we generate an animation that achieves the goals specified c* by the user? Scattered data interpolation

Scatter Data Interpolations

Preprocessed motions mi

Motion space: m

Control parameter space: s

How can we generate an animation that achieves the goals specified c* by the user? Scattered data interpolation

Scattered Data Interpolations

Motion space: m

Control parameter space: s

Many techniques for scattered data interp.:

- Local interpolation/regression [Kovar and Gleicher 04]- Radial basis functions [Rose et al 98]- Gaussian processes [Mukai and Kuriyama

05]

Results

Geostatistical motion interpolation [Mukai and Kuriyama 05]

- check youtube video [click here]

Discussions

Pros: + high-quality animation + good generalization: motion interpolation and

extrapolation + particularly suitable for high-level motion control

Cons - all examples must be structurally similar. - not compact: needs to retain original mocap data - not suitable for detailed kinematic motion control

such as such key frames - lack of planning schemes for task level control such

as moving from one point to another.

Data-driven animation

- Motion graphs

- Motion interpolations

- Statistical motion synthesis

Outline

Bayesian Motion Synthesis

Goal: Find the most likely motion x from control inputs c specified by the user

)|Pr(maxarg cxx

Bayesian Motion Synthesis

Goal: Find the most likely motion x from control inputs c specified by the user

)|Pr(maxarg cxx

)Pr()Pr()|Pr(maxarg cxxc

x

Bayesian Motion Synthesis

Goal: Find the most likely motion x from control inputs c specified by the user

)|Pr(maxarg cxx

)Pr()Pr()|Pr(maxarg cxxc

x

)Pr()|Pr(maxarg xxcx

Likelihood: How well motion matches control input? Motion priors: How

natural motion is?

Human motiondatabase

Human motionanalysis

Motionoptimization

Human motion prior

User-defined constraints

Generate natural human motion from a small set of user-defined constraints

Statistical Motion Synthesis

Statistical Dynamic Model

Character pose

Low-dimensional pose space

yt = C xt + D

Human motiondatabase

Human motionanalysis

Statistical dynamic model

xt = A1xt-1 +…+ Amxt-m+ B ut

Control input

Temporal prediction

Complexity of Statistical Dynamic Model

Character pose

Low-dimensional pose space

yt = C xt + D

Human motiondatabase

Human motionanalysis

Statistical dynamic model

xt = A1xt-1 +…+ Amxt-m+ B ut

Control input

Temporal prediction

dim(ut)

dim(xt)

Statistical Dynamic Model Learning

Human motiondatabase

Human motionanalysis

Character pose

Low-dimensional pose space

yt = C xt + D

xt = A1xt-1 +…+ Amxt-m+ B ut

Control input

Temporal prediction

Dynamic model matrices:

A1,…,Am, B, C, D

Statistical Dynamic Model Learning

Human motiondatabase

Human motionanalysis

Dynamic model matrices:

A1,…,Am, B, C, D

Dynamic system order:

m, dimensionality of xt and ut

Character pose

Low-dimensional pose space

yt = C xt + D

xt = A1xt-1 +…+ Amxt-m+ B ut

Control input

Temporal prediction

Reconstruction error

Full-body motion data

Xt = A1Xt-1+…+AmXt-m+B ut

Statistical linear dynamic model:

m = 3, dim(ut) = 4, error = 0.7 deg

Yt = C Xt + D

Generate natural motion from a small set of user-defined constraints

Overview: Offline Animation Control

Human motiondatabase

Human motionanalysis

Motionoptimization

Human motion prior

User-defined constraints

Constrained Motion Optimization

Human motiondatabase

Human motionanalysis

Motionoptimization

Human motion prior

User-defined constraints

Generate natural motion from a small set of user-defined constraints

Constrained Motion Optimization

Motionoptimization

Human motion prior

User-defined constraints

Smoothness term

yyE Tsmoothness

),,....,()(ln 1 tmtttt uxxxpup

2212 ttt CxCxCx

User-defined constraints:

Objective function:

Motion prior Motion smoothness

Sequential Quadratic Programming

t ttt

K

j

m

tiititjjtjux CxCxCxBuxAxuN 2

211

2

1, 2)),;(log(minarg

Constrained Motion Optimization

Mjcxf jj ,1)(

Results

Click here

Discussions

Pros: + good generalization and accurate motion control due

to the use of statistical models + compact: only needs to keep model parameters + suitable for kinematic motion control

Cons - often need to specify contact constraints across the

entire animation. - does not support task-level control (e.g., move from

one point to another) - better models are needed to match the synthesis

quality of motion graphs

Summary

The use of prerecorded motion data allows us to create high-quality controllable animation for human characters

Ideal data-driven animation techniques - High-quality: realistic animation without noticeable

visual artifacts - control accuracy: often require strong generalizability

to achieve new tasks. - control flexibility: support motion control in both

kinematics and task/behavior level. - scalability: scale up well to huge and heterogeneous

datasets. - compact: demand small or moderate memory sizes.