Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj
Go With The FlowA New Manifold Modeling and Learning Framework for Image Ensembles
Aswin C. SankaranarayananRice University
Sensor Data Deluge
Concise Models
• Efficient processing / compression requires concise representation
• Sparsity of an individual image
pixels largewaveletcoefficients
(blue = 0)
Concise Models
• Efficient processing / compression requires concise representation
• Our interest in this talk: Collections of images
Concise Models
• Our interest in this talk: Collections of image
parameterized by q \in Q
– translations of an object q: x-offset and y-offset
– rotations of a 3D object: q pitch, roll, yaw
– wedgeletsq: orientation and offset
Concise Models
• Our interest in this talk: Collections of image
parameterized by q \in Q
– translations of an object q: x-offset and y-offset
– rotations of a 3D object: q pitch, roll, yaw
– wedgeletsq: orientation and offset
• Image articulation manifold
Image Articulation Manifold• N-pixel images:
• K-dimensional articulation space
• Thenis a K-dimensional manifoldin the ambient space
• Very concise model
articulation parameter space
Smooth IAMs• N-pixel images:
• Local isometry: image distance parameter space distance
• Linear tangent spacesare close approximationlocally
articulation parameter space
Smooth IAMs• N-pixel images:
• Local isometry: image distance parameter space distance
• Linear tangent spacesare close approximationlocally
articulation parameter space
Ex: Manifold Learning
LLEISOMAPLEHEDiff. Geo …
• K=1rotation
Ex: Manifold Learning
• K=2rotation and scale
Theory/Practice Disconnect: Smoothness
• Practical image manifolds are not smooth!
• If images have sharp edges, then manifold is everywhere non-differentiable [Donoho and Grimes]
Tangent approximations ?Isometry ?
Theory/Practice Disconnect: Smoothness
• Practical image manifolds are not smooth!
• If images have sharp edges, then manifold is everywhere non-differentiable [Donoho and Grimes]
Tangent approximations ?Isometry ?
Failure of Tangent Plane Approx.
• Ex: cross-fading when synthesizing / interpolating images that should lie on manifold
Input Image Input Image
Geodesic Linear path
Failure of Local Isometry
• Ex: translation manifold
all blue imagesare equidistantfrom the red image
• Local isometry
– satisfied only when sampling is dense
0 20 40 60 80 1000
0.5
1
1.5
2
2.5
3
3.5
Translation in [px]
Euc
lidea
n di
stan
ce
Tools for manifold processing
Geodesics, exponential maps, log-maps, Riemannian metrics, Karcher means, …
Smooth diff manifold
Algebraic manifolds Data manifolds
LLE, kNN graphs
Point cloud model
The concept ofTransport operators
))(()( xfIxIref
Beyond point cloud model for image manifolds
refIfI
Example
Example: Translation
• 2D Translation manifold
barring boundary related issues
• Set of all transport operators =
• Beyond a point cloud model– Action of the articulation is more accurate and meaningful
1I
11101 )( ),()( xxfxIxI
0I
2R
Optical Flow
• Generalizing this idea: Pixel correspondances
• Idea:OF between two images is a natural and accurate transport operator
(Figures from Ce Liu’s optical flow page)
OFfrom I1 to I2
I1 and I2
Optical Flow Transport
0
1 2
IAM
2I1
I0I
Articulations
• Consider a reference imageand a K-dimensional articulation
• Collect optical flows fromto all images reachable by aK-dimensional articulation
Optical Flow Transport
0
1 2
IAM
OFM at 0I
2I1
I0I
Articulations
• Consider a reference imageand a K-dimensional articulation
• Collect optical flows fromto all images reachable by aK-dimensional articulation
• Theorem: Collection of OFs is a smooth, K-dimensional manifold(even if IAM is not smooth)
OFM is Smooth (Rotation)
00 30 60 30 60
-80 -60 -40 -20 0 20 40 60 800
50
100
150
200
Articulation in [ ]
Inte
nsity
-80 -60 -40 -20 0 20 40 60 80-20
-10
0
10
20
Articulation in [ ]
Opt
ical
flow
vx i
n pi
xels
Articulation θ in [⁰]
Inte
nsi
ty I(θ
)O
p. flow
v(θ
)Pixel intensityat 3 points
Flow (nearly linear)
Main results
• Local model at each
• Each point on the OFM defines a transport operator– Each transport operator maps
to one of its neighbors
• For a large class of articulations, OFMs are smooth and locally isometric– Traditional manifold processing
techniques work on OFMs
0
1 2
IAM
OFM at 0I
2I1
I0I
Articulations
0I
0I
Linking it all together
0
1 2
IAM
OFM at 0I
2I1
I0I
Articulations
0e
1e 2e
Nonlinear dim. reduction
The non-differentiablity does not dissappear --- it is embedded in the mapping from OFM to the IAM.
However, this is a known map
))(()( such that },{ xfIxIIfI refref
The Story So Far…
0
1 2
IAM
OFM at 0I
2I1
I0I
Articulations
Tangent space at 0I
Articulations
0
1 2
2I1
I0I IAM
Input Image Input Image
Geodesic Linear pathIAM
OFM
OFM Synthesis
ISOMAP embedding error for OFM and IAM
2D rotations
Reference image
Manifold Learning
Manifold Learning
-6 -4 -2 0 2 4 6-5
-4
-3
-2
-1
0
1
2
3
4
5Two-dimensional Isomap embedding (with neighborhood graph).
Embedding of OFM
2D rotations
Reference image
Data196 images of two bears moving linearlyand independently
IAM OFM
TaskFind low-dimensional embedding
OFM Manifold Learning
IAM OFM
Data196 images of a cup moving on a plane
Task 1Find low-dimensional embedding
Task 2Parameter estimation for new images(tracing an “R”)
OFM ML + Parameter Estimation
• Point on the manifold such that the sum of geodesic distances to every other point is minimized
• Important concept in nonlinear data modeling, compression, shape analysis [Srivastava et al]
Karcher Mean
10 images from an IAM
ground truth KM OFM KM linear KM
Manifold Charting
• Goal: build a generative model for an entire IAM/OFM based on a small number of base images
• Ex: cube rotating about axis. All cube images can be representing using 4 reference images + OFMs
• Many applications– selection of target templates for classification– “next-view” selection for adaptive sensing applications
Greedy charting
Optimal charting
0 90 180 180 90
IAM
Summary
• IAMs a useful concise model for many image processing problems involving image collections and multiple sensors/viewpoints
• But practical IAMs are non-differentiable– IAM-based algorithms have not lived up to their promise
• Optical flow manifolds (OFMs)– smooth even when IAM is not– OFM ~ nonlinear tangent space– support accurate image synthesis, learning, charting, …
• Barely discussed here: OF enables the safe extension of differential geometry concepts– Log/Exp maps, Karcher mean, parallel transport, …
Open Questions
• Our treatment is specific to image manifolds under
brightness constancy
• What are the natural transport operators for other data manifolds?
dsp.rice.edu
Related Work
• Analytic transport operators– transport operator has group structure [Xiao and Rao 07]
[Culpepper and Olshausen 09] [Miller and Younes 01] [Tuzel et al 08]
– non-linear analytics [Dollar et al 06]
– spatio-temporal manifolds [Li and Chellappa 10]
– shape manifolds [Klassen et al 04]
• Analytic approach limited to a small class of standard image transformations (ex: affine transformations, Lie groups)
• In contrast, OFM approach works reliably with real-world image samples (point clouds) and broader class of transformations
Limitations
• Brightness constancy– Optical flow is no longer meaningful
• Occlusion– Undefined pixel flow in theory, arbitrary flow estimates in
practice– Heuristics to deal with it
• Changing backgrounds etc.– Transport operator assumption too strict– Sparse correspondences ?
Open Questions
• Theorem: random measurements stably embed aK-dim manifoldwhp [B, Wakin, FOCM ’08]
• Q: Is there an analogousresult for OFMs?
Image Articulation Manifold
• Linear tangent space at is K-dimensional
– provides a mechanism to transport along manifold
– problem: since manifold is non-differentiable, tangent approximation is poor
• Our goal: replace tangent spacewith new transport operator that respects the nonlinearity of the imaging process
Tangent space at 0I
Articulations
0
1 2
2I1
I0I IAM
OFM Implementation details
Reference Image
Pairwise distances and embedding
-5 0 5
-4
-2
0
2
4
100 200 300 400
100
200
300
400
0 2 4 6 8 100
0.1
0.2
0.3
0.4
0.5
Res
idua
l var
ianc
e
Isomap dimensionality
-5 0 5
-4
-2
0
2
4100 200 300 400
100
200
300
400
0 2 4 6 8 100
0.1
0.2
0.3
0.4
0.5
Res
idua
l var
ianc
e
Isomap dimensionality
Flow Embedding
Occlusion
• Detect occlusion using forward-backward flow reasoning
• Remove occluded pixel computations
• Heuristic --- formal occlusion handling is hard
Occluded
History of Optical Flow
• Dark ages (<1985)– special cases solved– LBC an under-determined set of linear equations
• Horn and Schunk (1985) – Regularization term: smoothness prior on the flow
• Brox et al (2005)– shows that linearization of brightness constancy (BC) is
a bad assumption– develops optimization framework to handle BC directly
• Brox et al (2010), Black et al (2010), Liu et al (2010)– practical systems with reliable code