Video Textures

This is a very cool paper.

Why video texturing?

Static images are boring Video clips are finite in length We need something more interesting

What is video texturing?

The process of extracting an infinitely varying sequence of continuous images from a finite length video clip

For the purposes of creating backward compatible videos, it is sufficient to generate a single video clip with continuity between the final and beginning frames

The System

The System

Objectives Find points in the video between which transitions

can be made Smooth the transition as much as possible Rendering

An infinite random play A finite video clip using a looping sequence to

simulate infinite play

Representation

Probability Matrix Each entry indicates the probability of a certain

frame-to-frame transition being taken Good for dense set of possible transitions

Link Set Each entry indicates a link from one frame to

another and that transition Good for sparse set of possible transitions More typical

Techniques

Splitting the video into distinct regions

Techniques

Sprites

Extraction

First Equalize brightness Stabilize video Enhance in any other way desired

Compute the L2 distance to determine

similarity between frames Store them in the matrix:

Dij = ||Li-Lj||2

Extraction

Now transitions from frame i to frame j can be performed when Di+1,j is sufficiently small (i.e. similar)

Store the probability of these transitions in a probability matrix (or analogous structure):

Pij = exp(-Di+1,j/σ)

Then normalize each transition according to the rule:

ΣjPij = 1

The Problem

This creates smooth frame-to-frame transitions, but…

Preserving Dynamics

Smooth frame-to-frame transitions are not sufficient

Smooth motions must also be conveyed In other words

For a transition from i to j to be “good,” the frames i±m must be similar to the frames j±m

Preserving Dynamics Simply take the similarity matrix D and check

for similarities between frames to form the filtered matrix D’

D’ij = Σk(wkDi+k,j+k) where -m≤k≤m-1 And m is the number of frames to check on either side (1 or 2 typically)

Now compute probabilities based on D’

Preserving Dynamics

Graphically:

Preserving Dynamics

Much better, but…

Avoiding Dead Ends Now we have a dead end No problem…

Simply calculate the anticipated cost of transitioning from one frame to another D''ij = (D'ij)p + αΣkP''jkD''jk

High p favors the fewer good transitions Lower p favors the abundant poor transitions 0.99 <= α <= 0.999 makes a good alpha Alpha controls the affect that future transitions should have

P'' is now calculated with D''

Avoiding Dead Ends

Solving the interdependencies has a simple iterative solution… However, it’s slow to converge

So simplify the calculation D''ij = (D'ij)p + α·mink(P''jkD''jk) This assumes sigma approaches 0 (i.e. Pij approaches 1) for good transitions… only the best transition is considered

Avoiding Dead Ends See paper for details on iterative solution

Sequencing Random Play

Random (infinite) play Begin sequence at any transition before the last

non-zero probability transition while (1)

Determine transition after frame i by finding frame j such that Pij is greatest

Generates video textures that never repeat exactly

Sequencing Video Loops

Primitive loop Single transition from i to j

Must satisfy i ≥ j to guarantee loop Properties

Range = [j,i] Cost = D’ij

Length = i-j

Sequencing Video Loops

Compound Loop Combination of one or more primitive loops (or

compound loops) Each loop must overlap with at least one other Properties

Range = Uk range(Tk)

Cost = Σk cost(Tk)

Length = Σk length(Tk)

Sequencing Video Loops

Selecting Transition Sets Need to find the optimal loop for a given

sequence length This is just a simple DP algorithm

Still Sequencing Video Loops

Sequencing Video Loops

Sequencing Video Loops

Scheduling Now we have a set of transitions to use All we know, based on the properties of

compound loops is that the set represents a continuous range of frames

Must schedule the set into a coherent sequence of transitions

Sequencing Video Loops

Algorithm for Scheduling Primitive Loops 1. Schedule the transition at the end of the range

first 2. Remove transition from set – this may leave

one or more disjoint sets 3. Schedule the next transition from the first

disjoint set that begins after the beginning of the removed transition

4. Repeat 2-3 until no transitions remain

Rendering

Between transitions Cross fade Morph or de-ghosting

Between film regions Feathering

Questions?

Is “cyclified” a word?