View
229
Download
0
Tags:
Embed Size (px)
Citation preview
1Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Multidimensional scaling
© Alexander & Michael Bronstein, 2006-2009© Michael Bronstein, 2010tosca.cs.technion.ac.il/book
048921 Advanced topics in visionProcessing and Analysis of Geometric Shapes
EE Technion, Spring 2010
2Numerical Geometry of Non-Rigid Shapes Multidimensional scalingIf it doesn’t fit, you must acquit!If it doesn’t fit, you must acquit!
Image: Associated Press
3Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Metric model
Euclidean metric
Invariant to rigid
motion
EXTRINSIC GEOMETRY
Similarity = isometry w.r.t.
Shape = metric space
Extrinsic geometry is not invariant
under inelastic deformations
4Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Metric model
Geodesic metric
Invariant to inelastic
deformation
INTRINSIC GEOMETRY
Euclidean metric
Invariant to rigid
motion
EXTRINSIC GEOMETRY
Similarity = isometry w.r.t.
Shape = metric space
5Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Intrinsic vs. extrinsic similarity
INTRINSIC SIMILARITYEXTRINSIC SIMILARITY
Part of the same metric space Two different metric spaces
SOLUTION: Find a representation of and
in a common metric space
6Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Canonical forms
Isometric embedding
7Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Canonical form distance
Compute canonical formsEXTRINSIC SIMILARITY OF CANONICAL FORMS
INTRINSIC SIMILARITY
= INTRINSIC SIMILARITY
8Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Mapmaker’s problem
9Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Mapmaker’s problem
A sphere has non-zero curvature, therefore, it
is not isometric to the plane
(a consequence of Theorema egregium)
Karl Friedrich Gauss (1777-1855)
10Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Linial’s exampleA
C D
1 1
2
AC D
1 1
1 1
2C D
B
A
B
C
D
B
A
C D
1 1
B
1
Conclusion: generally, isometric embedding does not exist!
No contradiction to Nash embedding theorem: Nash guarantees that any
Riemannian structure can be realized as a length metric induced by
Euclidean metric. We try to realize it using restricted Euclidean metric
11Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Minimum distortion embedding
Minimum distortion embedding
12Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Multidimensional scaling
The problem is called multidimensional scaling (MDS)
Find an embedding into that distorts the distances the least by
solving the optimization problem
The function measuring the distortion of distances is called stress
where are the coordinates of the canonical form
13Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Stress
L2-stress
The stress is a function of the canonical form coordinates and
the distances
Lp-stress
L-stress (distortion)
14Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
- an matrix of canonical form coordinates (each row
corresponds to a point)
Matrix expression of L2-stress
Some notation:
Shorthand notation for Euclidean distances
Write the stress as
1 2
15Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Term 1
16Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Term 1
where is and matrix with elements
Matrices commute
under trace operator
17Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Term 2
For and
zero otherwise
18Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Term 2
where is and matrix-valued function with elements
19Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
variables
Non-linear
Non-convex (thus liable to local convergence)
Optimum defined up to Euclidean transformation
LS-MDS
Minimization of L2-stress is called least squares MDS (LS-MDS)
20Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Gradient of L2-stress
Recall exercise
in optimization
Quadratic in Nonlinear in Constant in
21Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Stress computation complexity:
Complexity
Steepest descent
with line search may be prohibitively expensive (requires multiple
evaluations of the stress and the gradient)
Stress gradient computation complexity:
22Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Observation
for all and
By Cauchy-Schwarz inequality
Equality is achieved for
Write it differently:
23Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Observation
Consider the nonlinear term in the stress
24Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Majorizing inequality
[de Leeuw, 1977]
Jan de Leeuw
Equality is achieved for
We have a quadratic majorizing function to
use in iterative majorization algorithm!
25Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Iterative majorization
Construct a majorizing function satisfying
.
Majorizing inequality: for all
is convex
26Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Iterative majorization
Start with some
Find such that
Update iterate
Increment iteration counter
Solution
Until
convergence
27Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Iterative majorization
The majorizing function
is quadratic!
Analytic expression for the minimim
Analytic expression for
pseudoinverse:
28Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
SMACOF algorithm
Start with some
Update iterate
Increment iteration counter
Solution
Until
convergence
The algorithm is called SMACOF (Scaling by Minimizing A COnvex
Function)
29Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
SMACOF is a constant-step gradient descent!
SMACOF algorithm vs steepest descent
Majorization guarantees monotonically decreasing sequence of
stress
values (untypical behavior for constant-step steepest descent)
No guarantee of global convergence
Iteration cost:
30Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
MATLAB® intermezzoSMACOF algorithm
Canonical forms
31Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Examples of canonical forms
Canonical forms
Near-isometric deformations of a shape
32Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Examples of canonical forms
Visualization of intrinsic similarity
33Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Weighted stress
where are some fixed weights
where is and matrix with elements
Matrix expression
SMACOF algorithm can be used to minimize weighted stress!
34Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Variations on the stress theme
Generic form of the stress
where is some norm
For example, gives the Lp-stress
The necessary condition for to be the minimizer of is
35Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Variations on the stress theme
Idea: minimize weighted stress instead of the generic stress and choose
the weights such that the two are equivalent
If the weights are selected as
the minimizers of and will coincide
Since is unknown in advance, iterate!
36Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Iteratively reweighted LS
Start with some and
Find
using as an initialization
Update weights
Increment iteration counter
Until
convergence
The algorithm is called Iteratively Reweighted Least Squares (IRLS)
37Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
L-stress
Optimization problem
can be written equivalently as a constrained problem
is called an artificial variable
38Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Complexity, bis
N=1000 points
MDS
N=200 points
MDS
x5 less pointsx25 lower complexity
39Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Multiresolution MDS
Bottom-up approach: solve coarse level MDS problem to initialize fine
level problem
Reduce complexity (less fine-level iterations)
Reduce the risk of local convergence (good initialization)
Can be performed on multiple resolution levels
40Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Solve coarse level problem
Interpolate
Two-resolution MDS
Solve finelevel problem
Grid
Data
Interpolation operator to transfer solution
from
coarse level to fine level
Can be represented as an sparse matrix
41Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Solve coarsest level problem
Interpolate
Multiresolution MDS
Solve -st level problem
Interpolate
Solve finestlevel problem
42Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Multiresolution MDS algorithm
Hierarchy of grids
Hierarchy of data
Interpolation operators
Start with coarsest-level initialization
Solve the l-th level MDS problem
using as initialization
Interpolate to next level
For
Solve finest level problem
using as initialization
43Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Top-down approach
Start with a fine-level initialization
Decimate fine-level initialization to the coarse grid
Solve the coarse-level MDS problem
using as initialization
Improve the fine-level solution propagating the coarse-level error
Typically
44Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
A problem
Fine level
Coarse level
45Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Correction
The fine- and the coarse-level minimizers do not coincide!
is called a residual
Force consistency of the coarse- and fine-level problems by introducing a
correction term into the stress
which gives the gradient
46Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Correction
Fine level
Coarse level
47Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Correction
In order to guarantee consistency, must satisfy
at
which gives the correction term
Verify: is the minimizer of coarse-level problem with correction,
Chain rule
48Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Modified stress
Another problem: the coarse grid problem
is unbounded for
Fix the translation ambiguity by adding a quadratic penalty to the stress
Can grow arbitrarily
is called the modified stress [Bronstein et al., 05]
The modified coarse grid problem is bounded
49Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Solve corrected coarse level problem
Two-grid MDS
Correct fine level solution
Decimate Interpolate
Iteratively improve the fine-level solution using coarse grid residual
External iteration is called cycle
50Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Solve corrected coarse level problem
Two-grid MDS
Correct fine level solution
Decimate Interpolate
Perform a few optimization iterations at fine level between cycles
(relaxation)
Relax
51Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Two-grid MDS algorithm
Start with fine-level initialization
Produce an improved fine-level solution by
making a few iterations on initialized with
Decimate fine-level solution
Correction:
Solve the coarse-level corrected MDS problem
using as initialization
Transfer residual:
Update iterate
Solution
Until
convergence
52Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Solve coarsest level problem
Interpolatecorrect
Multigrid MDS
Relax
Decimate
Relax
Decimate
Interpolatecorrect
Relax
Relax
Apply the two-grid algorithm recursively
Different cycles possible, e.g. V-cycle
53Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
MATLAB® intermezzoMultigrid MDS
54Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Convergence example
Time (sec)
Str
ess
Convergence of SMACOF and Multigrid MDS (N=2145)
55Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
How to accelerate convergence?
Let be a sequence of iterates produced by some
optimization algorithms converging to
Denote by the remainder
Construct a new sequence converging faster to
in the sense
56Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Vector extrapolation
Construct the new sequence as a transformation of iterates
Particular choice: linear combination
Question: how to choose the coefficients ?
Ideally, hence
57Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Reduced rank extrapolation (RRE)
Problem: depends on unknown
Eliminate this dependence by considering the difference
Result: solve the constrained linear system
which ideally should vanish.
of equations in variables
Typically hence the system is over-determined and must be
solved approximately using least-squares
58Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Start with some and
Generate iterates
using optimization on initialized by
Extrapolate from
Update
Increment iteration counter
Until
convergence
Optimization with vector extrapolation
59Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
Start with some and
Generate iterates
using optimization on initialized by
Extrapolate from
If
else
Increment iteration counter
Until
convergence
Safeguarded optimization with vector extrapolation
60Numerical Geometry of Non-Rigid Shapes Multidimensional scaling
MATLAB® intermezzoRRE MDS