1

Non-Linear Least Squares and Sparse Matrix Techniques:

Applications

Richard SzeliskiMicrosoft Research

UW-MSR Course onVision Algorithms

CSE/EE 577, 590CV, Spring 2004

5/4/2004 NLS and Sparse Matrix Techniques 2

Readings• R.Szeliski. Fast surface interpolation using hierarchical basis functions.

IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(6):513-528, June 1990.

• R. Szeliski and S. B. Kang. Recovering 3D shape and motion from image streams using nonlinear least squares. J. Vis. Comm. Image Representation, 5(1):10-28, March 1994. (Also available as CRL-93-3.)

• A. Levin and R. Szeliski, Visual Odometry and Map Correlation,CVPR’2004.

• P. Pérez, M. Gangnet, and A. Blake. Poisson Image Editing. SIGGRAPH’2003.

• L. Zhang, G. Dugas-Phocion, J.-S. Samson, and S. M. Seitz, Single view modeling of free-form scenes, J. Vis. Comp. Anim., 2002, vol. 13, no. 4, pp. 225-235.

• A. Agarwala, A. Hertzmann, D. H. Salesin, S. M. Seitz, Keyframe-Based Tracking for Rotoscoping and Animation, SIGGRAPH’2004.

5/4/2004 NLS and Sparse Matrix Techniques 3

Outline

Preconditioning• diagonal scaling, partial Cholesky

factorization• hierarchical basis functions (wavelets)• quadtree splines• 2D application:

height and normal interpolation(Single View Modeling)

5/4/2004 NLS and Sparse Matrix Techniques 4

Outline

Structure from motion• problems with size, linearity, conditioning• alternative parameterizations• uncertainty modeling

5/4/2004 NLS and Sparse Matrix Techniques 5

Outline

Visual Odometry and Map Correlation• fast visual localization• visual odometry↔map correlation• open problems

Preconditioned Conjugate Gradient

(See Shewchuk’s TR)

2

5/4/2004 NLS and Sparse Matrix Techniques 7

Conditioning

Conjugate gradient’s performance depends on the condition number of the Hessian

5/4/2004 NLS and Sparse Matrix Techniques 8

Transformation of variables

Replace original unknowns x with transformed variables x = ETx

The new condition number depends onκ(M-1A) with M = EET

^

5/4/2004 NLS and Sparse Matrix Techniques 9

Untransformed Precond. CG

Don’t perform transform of variables explicitlyInstead, “rotate” the residual vector, M-1r

5/4/2004 NLS and Sparse Matrix Techniques 10

Diagonal and Cholesky precond.

Simple preconditioner (almost for free) is diagonal preconditioning, M = diag(A)

Better preconditioner is incomplete Choleskyfactorization, M = LLT, A ≈ LLT

• minimize fill-in (non-zero entries)• not always stable (Shewchuk)Hierarchical bases [Szeliski, PAMI’90]

5/4/2004 NLS and Sparse Matrix Techniques 11

Hierarchical Basis Functions

• Proposed by Yserentant[1986]

• Use a multi-level(pyramid-like) basis for system

x = Sy, S = S1… SL-1

5/4/2004 NLS and Sparse Matrix Techniques 12

2-D Hierarchical basis

Like a pyramid, but complete (no extra variables)

3

5/4/2004 NLS and Sparse Matrix Techniques 13

Surface interpolation example

5/4/2004 NLS and Sparse Matrix Techniques 14

Surface interpolation - performance

5/4/2004 NLS and Sparse Matrix Techniques 15

Surface interpolation - performance

5/4/2004 NLS and Sparse Matrix Techniques 16

Surface interpolation - performance

5/4/2004 NLS and Sparse Matrix Techniques 17

Eigenvalue analysis

5/4/2004 NLS and Sparse Matrix Techniques 18

Comparison with multigrid

Solve a series of nested problems at different resolutions, using injection and prolongationto move between levels [Briggs, 1987]

(Personal experience): does not seem to work well on inhomogeneous problems (as opposed to very fine grid refinement)

4

5/4/2004 NLS and Sparse Matrix Techniques 19

Open issues

How many levels to use?Which order interpolant?Adaptation to data density and continuity• intuition: when data terms dominate, want

nodal basis; when smoothness, want hierarchical

5/4/2004 NLS and Sparse Matrix Techniques 20

Quadtree splines

Only populate (estimate) a subset of mesh

[Szeliski & Shum, PAMI’96]

5/4/2004 NLS and Sparse Matrix Techniques 21

Quadtree spline

Equivalent to sparse hierarchical basis

5/4/2004 NLS and Sparse Matrix Techniques 22

Quadtree splines - application

Discontinuous optic flow

Discontinuous single view modeling (next)

Single view modelingof free-form scenes

L. Zhang, G. Dugas-Phocion, J.-S. Samson, and S. M. Seitz

5/4/2004 NLS and Sparse Matrix Techniques 24

Problem

Extract a 3D model from a single image

5

5/4/2004 NLS and Sparse Matrix Techniques 25

Normal constraints

Fix the local orientation of the surface

• Dy f(P) =

• Dx f(P) =

P : point where we set the

normal constraint

α

β

γ εaf(ε) – f(γ)

a

f(β) – f(α)a

5/4/2004 NLS and Sparse Matrix Techniques 26

Sample deformations

5/4/2004 NLS and Sparse Matrix Techniques 27

Speed up the solver

Wavelet transformation (from coefficient to height)

1D Convolution 2D Convolution

hI = (hA + hB) / 2 + cI

5/4/2004 NLS and Sparse Matrix Techniques 28

Speed up the solver

What happens with discontinuities ?

Instead of : hI = (hA + hB) / 2 + cI

We get :hI = hB + cI

1D Convolution

5/4/2004 NLS and Sparse Matrix Techniques 29

Advantage of discontinuities

5/4/2004 NLS and Sparse Matrix Techniques 30

User specifiedUser specified High curvatureHigh curvatureAlong a curveAlong a curve

Hierarchical structure

Ways of changing the resolution.

Along a curve High curvature

6

5/4/2004 NLS and Sparse Matrix Techniques 31

Final results

Structure from motion

5/4/2004 NLS and Sparse Matrix Techniques 33

Problem areas

Problems with size, linearity, conditioning• reduce the number of variables

[Shum, Ke, Zhang]• partition the cameras / data [Steedly et al.]Uncertainty modelingAlternative parameterizations

5/4/2004 NLS and Sparse Matrix Techniques 34

Uncertainty modeling

Bas-relief ambiguity:• correlation between depth and

motion/rotation

5/4/2004 NLS and Sparse Matrix Techniques 35

Uncertainty modeling

5/4/2004 NLS and Sparse Matrix Techniques 36

Alternative projection

Use object-centered representation

7

Video-Based Rendering

5/4/2004 NLS and Sparse Matrix Techniques 38

Video-Based Virtual Tours

Move camera along a rail (“dolly track”) and play back a 360° video

Applications:• Homes and architecture• Outdoor locations

(tourist destinations)

5/4/2004 NLS and Sparse Matrix Techniques 39

360° video camera

OmniCam (six-camera head)5/4/2004 NLS and Sparse Matrix Techniques 40

OmniCam

Built by Point Grey Research (Ladybug)

Six camera head

Portable hard drives, fiber-optic link

Resolution per image: 1024 x 768

FOV: ~100o x ~80o

Acquisition speed: 15 fps uncompressed

5/4/2004 NLS and Sparse Matrix Techniques 41

Acquisition platforms

Robotic cart

Wearable

5/4/2004 NLS and Sparse Matrix Techniques 42

Stitching

(Only 4 of 6 images shown here)

8

5/4/2004 NLS and Sparse Matrix Techniques 43

Feature tracking for stabilization

Points tracked in all frames in all 6 cameras

Edges tracked in all frames in all 6 cameras5/4/2004 NLS and Sparse Matrix Techniques 44

Stabilization

Before motion stabilization After motion stabilization

Align frame-

to-frameand

distribute∆heading

5/4/2004 NLS and Sparse Matrix Techniques 45

Head-mounted outdoor scene acquisitionMap controlLocalized audio for a richer experienceComplex path navigation

(greater freedom ofmotion)

Outdoor Garden Tour

5/4/2004 NLS and Sparse Matrix Techniques 46

Map control• The garden map

• Drawing the acquisition path

• Mapping video framespositions on map

• Placing sound sources(background and dynamic)

• Output is a descriptiveXML file

5/4/2004 NLS and Sparse Matrix Techniques 47

Bifurcation handling

Hypothesize, align, choose best pair as bifurcation

Choice of path depends on current heading and bifurcation point orientation

Current heading

5/4/2004 NLS and Sparse Matrix Techniques 48

Bifurcation handling

Hypothesize, align, choose best pair as bifurcation

Choice of path depends on current heading and bifurcation point orientation

Current heading

9

Demo 1A walk in Bellevue Botanical Garden

Demo 2A High-End Home

5/4/2004 NLS and Sparse Matrix Techniques 51

Future directions

Towards true 3D:• better compression• object interaction (e.g., vase on table)• object compositing (video game)• more general motion (“true” IBR)• automate generation of bifurcation points and

map control (UI)

Visual Odometry andMap Correlation

Anat Levin and Richard SzeliskiCVPR’2004

5/4/2004 NLS and Sparse Matrix Techniques 53

Map Correlation

Four stages:1. Overlap detection (path bifurcations, better

egomotion estimation)2. Map correlation: [temporally] align hand-drawn

map with video-based ego-motion estimates 3. Align ego-motion to map4. Optimize camera position from both visual and

map data

5/4/2004 NLS and Sparse Matrix Techniques 54

Map Correlation

100

157

178

267

479552

564

670693

794813

879

119 43

211

430299

313611

637

719740

282

239

−50 0 50 100 150 200 250−50

0

50

100

150

200

250

300

?

10

5/4/2004 NLS and Sparse Matrix Techniques 55

Map Correlation

329241

211

164

1

4513794286

355417374

434

−30 −20 −10 0 10 20 30 40 50 600

20

40

60

80

100

120 ?

5/4/2004 NLS and Sparse Matrix Techniques 56

1. Overlap detection

−60 −40 −20 0 20 40 60 80 100−50

0

50

100

150

200

1

1943

100

157178

211239

267

282299

313

430 479552

564

611637

670693719

740

794

813879

50 100 150 200 250 300 350 400 450 500−500

−450

−400

−350

−300

−250

−200

−150

−100

1

1943

100

157178

211239

267

282

299

313

430 479

552

564

611

637 670

693719

740

794 813

879

100

157

178

267

479552

564

670693

794813

879

119 43

211

430299

313611

637

719740

282

239

5/4/2004 NLS and Sparse Matrix Techniques 57

Overlap detection

0 100 200 300 400 500 600 700−700

−600

−500

−400

−300

−200

−100

0

329 241 211

164

1

4513794286

355417374

434

−30 −20 −10 0 10 20 30 40 50 600

20

40

60

80

100

120

329 241

211

164

1

45137

94

286

355417

374434

329241

211

164

1

4513794286

355417374

434

5/4/2004 NLS and Sparse Matrix Techniques 58

Overlap detection •Stage 1: color histograms pruning •Any ideas?

5/4/2004 NLS and Sparse Matrix Techniques 59

Overlap detection •Stage 1: color histograms pruning

(1,100)

(19,157)(43,178)

(211,267)

(430,479)

(299,552)

(313,564)

(611,670)(637,693)

(719,794)(740,813)

(282,879)

(239,239)

100 200 300 400 500 600 700 800

100

200

300

400

500

600

700

800

100

157

178

267

479552

564

670693

794813

879

119 43

211

430299

313611

637

719740

282

239

5/4/2004 NLS and Sparse Matrix Techniques 60

(45,137)

(45,286) (137,286)

(94,94)

(286,286)

(241,329)

(374,434)(355,417)

50 100 150 200 250 300 350 400 450

50

100

150

200

250

300

350

400

450

Overlap detection •Stage 1: color histograms pruning

329241

211

164

1

4513794286

355417374

434

11

5/4/2004 NLS and Sparse Matrix Techniques 61

Overlap detection •Stage 2: moment matching

I(u) ↔I(Ru)∫uI(Ru)du ↔RT ∫ vI(v)dv

use different Image functions (color histogram bins)

5/4/2004 NLS and Sparse Matrix Techniques 62

Overlap detection Stage 2: For each (I1(u),I2(u)) with similar color histograms:

RANSAC computation of R based on 1st order moments

5/4/2004 NLS and Sparse Matrix Techniques 63

Overlap detection Stage 2: For each (I1(u),I2(u)) with similar color histograms:

RANSAC computation of R based on 1st order moments

5/4/2004 NLS and Sparse Matrix Techniques 64

Overlap detection •Stage 2: moments matching

(1,100)

(19,157)(43,178)

(211,267)

(430,479)

(299,552)

(313,564)

(611,670)(637,693)

(719,794)(740,813)

(282,879)

(239,239)

100 200 300 400 500 600 700 800

100

200

300

400

500

600

700

800

100

157

178

267

479552

564

670693

794813

879

119 43

211

430299

313611

637

719740

282

239

5/4/2004 NLS and Sparse Matrix Techniques 65

Overlap detection •Stage 2: moments matching

(45,137)

(45,286) (137,286)

(94,94)

(286,286)

(241,329)

(374,434)(355,417)

50 100 150 200 250 300 350 400 450

50

100

150

200

250

300

350

400

450

329241

211

164

1

4513794286

355417374

434

5/4/2004 NLS and Sparse Matrix Techniques 66

Overlap detection •Stage 3:Features matching & epipolar constraints

(1,100)

(19,157)(43,178)

(211,267)

(430,479)

(299,552)

(313,564)

(611,670)(637,693)

(719,794)(740,813)

(282,879)

(239,239)

100

200

300

400

500

600

700

800

100

157

178

267

479552

564

670693

794813

879

119 43

211

430299

313611

637

719740

282

239

12

5/4/2004 NLS and Sparse Matrix Techniques 67

(45,137)

(45,286) (137,286)

(94,94)

(286,286)

(241,329)

(374,434)(355,417)

50 100 150 200 250 300 350 400

50

100

150

200

250

300

350

400

450

Overlap detection •Stage 3:Feature matching & epipolar constraints

329241

211

164

1

4513794286

355417374

434

5/4/2004 NLS and Sparse Matrix Techniques 68

Overlap detection 1 2 3

(45,137)

(45,286) (137,286)

(94,94)

(286,286)

(241,329)

(374,434)(355,417)

50 100 150 200 250 300 350 400 450

50

100

150

200

250

300

350

400

450

(45,137)

(45,286) (137,286)

(94,94)

(286,286)

(241,329)

(374,434)(355,417)

50 100 150 200 250 300 350 400 450

50

100

150

200

250

300

350

400

450

(45,137)

(45,286) (137,286)

(94,94)

(286,286)

(241,329)

(374,434)(355,417)

50 100 150 200 250 300 350 400

50

100

150

200

250

300

350

400

450

329241

211

1644513794286

355417374

434

5/4/2004 NLS and Sparse Matrix Techniques 69

Overlap detection

(1,100)

(19,157)(43,178)

(211,267)

(430,479)

(299,552)

(313,564)

(611,670)(637,693)

(719,794)(740,813)

(282,879)

(239,239)

100 200 300 400 500 600 700 800

100

200

300

400

500

600

700

800

(1,100)

(19,157)(43,178)

(211,267)

(430,479)

(299,552)

(313,564)

(611,670)(637,693)

(719,794)(740,813)

(282,879)

(239,239)

100 200 300 400 500 600 700 800

100

200

300

400

500

600

700

800

(1,100)

(19,157)(43,178)

(211,267)

(430,479)

(299,552)

(313,564)

(611,670)(637,693)

(719,794)(740,813)

(282,879)

(239,239)

100 200 300 400 500 600 700 800

100

200

300

400

500

600

700

800

1 2 3

100

157

178

267

479552

564

670693

794813

879

119 43

211

430299

313611

637

719740

282

2395/4/2004 NLS and Sparse Matrix Techniques 70

2. Map correlation

−40 −20 0 20 40 60 80 100−20

0

20

40

60

80

100

120

140

160

180

1

2045

67

99

169 181

213240

271281

297

319

423434 477485

550

571

589

615

637

674692721

739

798 813

353

368 391

866

56126

515526 532

matches− ego motion

50 100 150 200 250 300 350 400 450 500−500

−450

−400

−350

−300

−250

−200

−150

−100

1

2045

6799

169181

213240

271281

297

319

423434 477

485550

571

589615

637 674

692

721

739

798 813

353

368 391

866

56126

515526

532

matches− map

5/4/2004 NLS and Sparse Matrix Techniques 71

•Tracks visible only other a short time window-> ego motion estimation tends to drift over time

•Unknown velocities

Ego-motion challenges

−50 0 50 100 150 200 250−50

0

50

100

150

200

250

300

5/4/2004 NLS and Sparse Matrix Techniques 72

Ego motion <-> map matching

Find:

F:{1,…,m} → {1,…,n}

Map points

Sequence frames

Any ideas?

13

5/4/2004 NLS and Sparse Matrix Techniques 73

Ego motion ↔ map matchingFind: F:{1,…,m} → {1,…,n}

A “good” F:

•Monotonic and smooth

•Minimize local orientation differences

• close on the map →Identified as “same location” by the previous stage

),( ji mm ),( ji FF

5/4/2004 NLS and Sparse Matrix Techniques 74

Graphical model representationNodes- discretized map points

Mapped into sequence frames

50 100 150 200 250 300 350 400 450 500−500

−450

−400

−350

−300

−250

−200

−150

−100

5/4/2004 NLS and Sparse Matrix Techniques 75

Graphical model representation

Ego-motion Map

•F Monotonic and smooth

5/4/2004 NLS and Sparse Matrix Techniques 76

Graphical model representation

Ego-motion Map

•F Monotonic and smooth

5/4/2004 NLS and Sparse Matrix Techniques 77

Graphical model representation

Ego-motion Map

•F Monotonic and smooth

5/4/2004 NLS and Sparse Matrix Techniques 78

Graphical model representation

50 100 150 200 250 300 350 400 450 500−500

−450

−400

−350

−300

−250

−200

−150

−100

•F Monotonic and smooth

ψ1i,i+1 = exp(-║Fi-Fi+1║)[Fi≤Fi+1]

14

5/4/2004 NLS and Sparse Matrix Techniques 79

Graphical model representation•F Minimize local orientation differences

ψ2i,i+ρ = exp(-║[α(i)-α(i+ρ)║]- [α(Fi)-α(Fi+ρ)║]

Ego-motion Map

5/4/2004 NLS and Sparse Matrix Techniques 80

Graphical model representation

50 100 150 200 250 300 350 400 450 500−500

−450

−400

−350

−300

−250

−200

−150

−100

•F Minimize local orientation differences

5/4/2004 NLS and Sparse Matrix Techniques 81

Graphical model representation

50 100 150 200 250 300 350 400 450 500−500

−450

−400

−350

−300

−250

−200

−150

−100

• (mi,mj) close on the map → (Fi,Fj)

ψ1i,j = iff (Fi,Fj) identified as “same location”

5/4/2004 NLS and Sparse Matrix Techniques 82

Graphical model representation• (mi,mj) close on the map → (Fi,Fj)

ψ1i,j = iff (Fi,Fj) identified as “same location”

(1,100)

(19,157)(43,178)

(211,267)

(430,479)

(299,552)

(313,564)

(611,670)(637,693)

(719,794)(740,813)

(282,879)

(239,239)

100 200 300 400 500 600 700 800

100

200

300

400

500

600

700

800

5/4/2004 NLS and Sparse Matrix Techniques 83

Graphical model representation

Use loopy belief propagation to find assignment maximizing:

50 100 150 200 250 300 350 400 450 500−500

−450

−400

−350

−300

−250

−200

−150

−100

50 100 150 200 250 300 350 400 450 500−500

−450

−400

−350

−300

−250

−200

−150

−100

50 100 150 200 250 300 350 400 450 500−500

−450

−400

−350

−300

−250

−200

−150

−100

How do we find the best assignment?

5/4/2004 NLS and Sparse Matrix Techniques 84

Ego motion ↔ map matching

−40 −20 0 20 40 60 80 100−20

0

20

40

60

80

100

120

140

160

180

1

2045

67

99

169 181

213240

271281

297

319

423434 477485

550

571

589

615

637

674692721

739

798 813

353

368 391

866

56126

515526 532

matches− ego motion

50 100 150 200 250 300 350 400 450 500−500

−450

−400

−350

−300

−250

−200

−150

−100

1

2045

6799

169181

213240

271281

297

319

423434 477

485550

571

589615

637 674

692

721

739

798 813

353

368 391

866

56126

515526

532

matches− map

15

5/4/2004 NLS and Sparse Matrix Techniques 85

Ego motion ↔ map matching

−50 −40 −30 −20 −10 0 10 20 30 400

20

40

60

80

100

120

140

327238

208

162

2

44135

92284

352415

371434

matches− ego motion

0 100 200 300 400 500 600 700−700

−600

−500

−400

−300

−200

−100

0

327 238208

162

2

4413592284

352415371

434

matches− map

5/4/2004 NLS and Sparse Matrix Techniques 86

Transforming ego motion → map

Obtain initial guess for motion parameters-minimizing distance to map while preserving ego-motion high frequencies.

−40 −20 0 20 40 60 80 100−20

0

20

40

60

80

100

120

140

160

180

1

2045

67

99

169 181

213240

271281

297

319

423434 477485

550

571

589

615

637

674692721

739

798 813

353

368 391

866

56126

515526 532

matches− ego motion

50 100 150 200 250 300 350 400 450 500−500

−450

−400

−350

−300

−250

−200

−150

−100

1

2045

6799

169181

213240

271281

297

319

423434 477

485550

571

589615

637 674

692

721

739

798 813

353

368 391

866

56126

515526

532

matches− map

5/4/2004 NLS and Sparse Matrix Techniques 87

Transforming ego motion → map

Local Euclidian transformations:

•Minimizing distance to map

•Agree on concatenation points

•Respect overlapping constraints

T1T2

5/4/2004 NLS and Sparse Matrix Techniques 88

Transforming ego motion → map

50 100 150 200 250 300 350 400 450 500−500

−450

−400

−350

−300

−250

−200

−150

−100

5/4/2004 NLS and Sparse Matrix Techniques 89

Transforming ego motion → map

0 100 200 300 400 500 600 700−700

−600

−500

−400

−300

−200

−100

0

5/4/2004 NLS and Sparse Matrix Techniques 90

Bundle adjustment optimization

120 140 160 180 200 220 240 260 280 300−500

−450

−400

−350

−300

−250Input:

•Visual features tracks

•Initial guess for camera positions

Non linear refining of estimation

Input

Optimized

16

5/4/2004 NLS and Sparse Matrix Techniques 91

Bundle adjustment optimization

Lessons learned:

•poor linearization

•numerical conditioning

•may need alternate representation and/or optimization algorithm

5/4/2004 NLS and Sparse Matrix Techniques 92

Match Refining

Input:

Optimization

Refining match

5/4/2004 NLS and Sparse Matrix Techniques 93

1.Fast overlap detection

2.Rough ego-motion <-> map correlation based on rotations (orientation)

3.Local non-linear optimization

Visual Odometry & Map Correlation

Questions?