View
218
Download
0
Category
Tags:
Preview:
Citation preview
Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj
Go With The FlowA New Manifold Modeling and Learning Framework for Image Ensembles
Aswin C. SankaranarayananRice University
Sensor Data Deluge
Concise Models
• Efficient processing / compression requires concise representation
• Sparsity of an individual image
pixels largewaveletcoefficients
(blue = 0)
Concise Models
• Efficient processing / compression requires concise representation
• Our interest in this talk: Collections of images
Concise Models
• Our interest in this talk: Collections of image
parameterized by q \in Q
– translations of an object q: x-offset and y-offset
– rotations of a 3D object: q pitch, roll, yaw
– wedgeletsq: orientation and offset
Concise Models
• Our interest in this talk: Collections of image
parameterized by q \in Q
– translations of an object q: x-offset and y-offset
– rotations of a 3D object: q pitch, roll, yaw
– wedgeletsq: orientation and offset
• Image articulation manifold
Image Articulation Manifold• N-pixel images:
• K-dimensional articulation space
• Thenis a K-dimensional manifoldin the ambient space
• Very concise model
articulation parameter space
Smooth IAMs• N-pixel images:
• Local isometry: image distance parameter space distance
• Linear tangent spacesare close approximationlocally
articulation parameter space
Smooth IAMs• N-pixel images:
• Local isometry: image distance parameter space distance
• Linear tangent spacesare close approximationlocally
articulation parameter space
Ex: Manifold Learning
LLEISOMAPLEHEDiff. Geo …
• K=1rotation
Ex: Manifold Learning
• K=2rotation and scale
Theory/Practice Disconnect: Smoothness
• Practical image manifolds are not smooth!
• If images have sharp edges, then manifold is everywhere non-differentiable [Donoho and Grimes]
Tangent approximations ?Isometry ?
Theory/Practice Disconnect: Smoothness
• Practical image manifolds are not smooth!
• If images have sharp edges, then manifold is everywhere non-differentiable [Donoho and Grimes]
Tangent approximations ?Isometry ?
Failure of Tangent Plane Approx.
• Ex: cross-fading when synthesizing / interpolating images that should lie on manifold
Input Image Input Image
Geodesic Linear path
Failure of Local Isometry
• Ex: translation manifold
all blue imagesare equidistantfrom the red image
• Local isometry
– satisfied only when sampling is dense
0 20 40 60 80 1000
0.5
1
1.5
2
2.5
3
3.5
Translation in [px]
Euc
lidea
n di
stan
ce
Tools for manifold processing
Geodesics, exponential maps, log-maps, Riemannian metrics, Karcher means, …
Smooth diff manifold
Algebraic manifolds Data manifolds
LLE, kNN graphs
Point cloud model
The concept ofTransport operators
))(()( xfIxIref
Beyond point cloud model for image manifolds
refIfI
Example
Example: Translation
• 2D Translation manifold
barring boundary related issues
• Set of all transport operators =
• Beyond a point cloud model– Action of the articulation is more accurate and meaningful
1I
11101 )( ),()( xxfxIxI
0I
2R
Optical Flow
• Generalizing this idea: Pixel correspondances
• Idea:OF between two images is a natural and accurate transport operator
(Figures from Ce Liu’s optical flow page)
OFfrom I1 to I2
I1 and I2
Optical Flow Transport
0
1 2
IAM
2I1
I0I
Articulations
• Consider a reference imageand a K-dimensional articulation
• Collect optical flows fromto all images reachable by aK-dimensional articulation
Optical Flow Transport
0
1 2
IAM
OFM at 0I
2I1
I0I
Articulations
• Consider a reference imageand a K-dimensional articulation
• Collect optical flows fromto all images reachable by aK-dimensional articulation
• Theorem: Collection of OFs is a smooth, K-dimensional manifold(even if IAM is not smooth)
OFM is Smooth (Rotation)
00 30 60 30 60
-80 -60 -40 -20 0 20 40 60 800
50
100
150
200
Articulation in [ ]
Inte
nsity
-80 -60 -40 -20 0 20 40 60 80-20
-10
0
10
20
Articulation in [ ]
Opt
ical
flow
vx i
n pi
xels
Articulation θ in [⁰]
Inte
nsi
ty I(θ
)O
p. flow
v(θ
)Pixel intensityat 3 points
Flow (nearly linear)
Main results
• Local model at each
• Each point on the OFM defines a transport operator– Each transport operator maps
to one of its neighbors
• For a large class of articulations, OFMs are smooth and locally isometric– Traditional manifold processing
techniques work on OFMs
0
1 2
IAM
OFM at 0I
2I1
I0I
Articulations
0I
0I
Linking it all together
0
1 2
IAM
OFM at 0I
2I1
I0I
Articulations
0e
1e 2e
Nonlinear dim. reduction
The non-differentiablity does not dissappear --- it is embedded in the mapping from OFM to the IAM.
However, this is a known map
))(()( such that },{ xfIxIIfI refref
The Story So Far…
0
1 2
IAM
OFM at 0I
2I1
I0I
Articulations
Tangent space at 0I
Articulations
0
1 2
2I1
I0I IAM
Input Image Input Image
Geodesic Linear pathIAM
OFM
OFM Synthesis
ISOMAP embedding error for OFM and IAM
2D rotations
Reference image
Manifold Learning
Manifold Learning
-6 -4 -2 0 2 4 6-5
-4
-3
-2
-1
0
1
2
3
4
5Two-dimensional Isomap embedding (with neighborhood graph).
Embedding of OFM
2D rotations
Reference image
Data196 images of two bears moving linearlyand independently
IAM OFM
TaskFind low-dimensional embedding
OFM Manifold Learning
IAM OFM
Data196 images of a cup moving on a plane
Task 1Find low-dimensional embedding
Task 2Parameter estimation for new images(tracing an “R”)
OFM ML + Parameter Estimation
• Point on the manifold such that the sum of geodesic distances to every other point is minimized
• Important concept in nonlinear data modeling, compression, shape analysis [Srivastava et al]
Karcher Mean
10 images from an IAM
ground truth KM OFM KM linear KM
Manifold Charting
• Goal: build a generative model for an entire IAM/OFM based on a small number of base images
• Ex: cube rotating about axis. All cube images can be representing using 4 reference images + OFMs
• Many applications– selection of target templates for classification– “next-view” selection for adaptive sensing applications
Greedy charting
Optimal charting
0 90 180 180 90
IAM
Summary
• IAMs a useful concise model for many image processing problems involving image collections and multiple sensors/viewpoints
• But practical IAMs are non-differentiable– IAM-based algorithms have not lived up to their promise
• Optical flow manifolds (OFMs)– smooth even when IAM is not– OFM ~ nonlinear tangent space– support accurate image synthesis, learning, charting, …
• Barely discussed here: OF enables the safe extension of differential geometry concepts– Log/Exp maps, Karcher mean, parallel transport, …
Open Questions
• Our treatment is specific to image manifolds under
brightness constancy
• What are the natural transport operators for other data manifolds?
dsp.rice.edu
Related Work
• Analytic transport operators– transport operator has group structure [Xiao and Rao 07]
[Culpepper and Olshausen 09] [Miller and Younes 01] [Tuzel et al 08]
– non-linear analytics [Dollar et al 06]
– spatio-temporal manifolds [Li and Chellappa 10]
– shape manifolds [Klassen et al 04]
• Analytic approach limited to a small class of standard image transformations (ex: affine transformations, Lie groups)
• In contrast, OFM approach works reliably with real-world image samples (point clouds) and broader class of transformations
Limitations
• Brightness constancy– Optical flow is no longer meaningful
• Occlusion– Undefined pixel flow in theory, arbitrary flow estimates in
practice– Heuristics to deal with it
• Changing backgrounds etc.– Transport operator assumption too strict– Sparse correspondences ?
Open Questions
• Theorem: random measurements stably embed aK-dim manifoldwhp [B, Wakin, FOCM ’08]
• Q: Is there an analogousresult for OFMs?
Image Articulation Manifold
• Linear tangent space at is K-dimensional
– provides a mechanism to transport along manifold
– problem: since manifold is non-differentiable, tangent approximation is poor
• Our goal: replace tangent spacewith new transport operator that respects the nonlinearity of the imaging process
Tangent space at 0I
Articulations
0
1 2
2I1
I0I IAM
OFM Implementation details
Reference Image
Pairwise distances and embedding
-5 0 5
-4
-2
0
2
4
100 200 300 400
100
200
300
400
0 2 4 6 8 100
0.1
0.2
0.3
0.4
0.5
Res
idua
l var
ianc
e
Isomap dimensionality
-5 0 5
-4
-2
0
2
4100 200 300 400
100
200
300
400
0 2 4 6 8 100
0.1
0.2
0.3
0.4
0.5
Res
idua
l var
ianc
e
Isomap dimensionality
Flow Embedding
Occlusion
• Detect occlusion using forward-backward flow reasoning
• Remove occluded pixel computations
• Heuristic --- formal occlusion handling is hard
Occluded
History of Optical Flow
• Dark ages (<1985)– special cases solved– LBC an under-determined set of linear equations
• Horn and Schunk (1985) – Regularization term: smoothness prior on the flow
• Brox et al (2005)– shows that linearization of brightness constancy (BC) is
a bad assumption– develops optimization framework to handle BC directly
• Brox et al (2010), Black et al (2010), Liu et al (2010)– practical systems with reliable code
Recommended