Computational Optical Imaging - Optique Numerique...

Ivo Ihrke / Autumn 2015

Computational Optical Imaging -Optique Numerique

-- Single and Multiple View Geometry, Stereo matching --

Autumn 2015

Ivo Ihrke

with slides by Thorsten Thormaehlen

Reminder: Feature Detection and Matching

Wide-Baseline-Matching

SIFT = Scale Invariant Feature Transform David G. Lowe: “Distinctive image features from scale-invariant keypoints” (IJCV 2004)

http://www.cs.ubc.ca/~lowe/keypoints/

Suited for wide-baseline matching

Invariance to changes in illumination, scale, and rotation

No gradient approach, but compares feature description of all candidates

Many applications

Wide-baseline Matching with SIFT

Application example: Mosaic generation

Application example: Object recognition

Stereo pair with super-imposed SIFT features

SIFT: Keypoint localization

Arrow size indicates scale

729 out of 832 feature are left after contrast thresholding

(try cross-eye fusion to identify the outliers :)

SIFT: Descriptor matching

VLFeat (http://www.vlfeat.org/index.html)

Implements SIFT and other modern features

C-implementation with MATLAB bindings

─ Win/Linux

Good tutorials on parameter selection

Software

Towards Multiple Views and Self-Calibration

-- Two-View Geometry and Basic Stereo --

Fundamental Matrix (F-Matrix):

Fundamental Matrix

Epipolar Geometry in our example

F is a rank 2 homogeneous matrix with 7 degrees of freedom

Epipolar lines

Epipoles

Fundamental Matrix

F can be computed from camera matrices

General projective cameras:

with and

Canonical cameras not at infinity and

Fundamental Matrix

Typical left and right image with parallel optical axes and only horizontally displaced

Use Case – Stereo Matching

[implementation by Rohit Singh and Mitul Sara]

Left image Right image

Epipolar lines are parallel lines if optical axes are parallel

Epipolar Geometry of a Stereo Pair

Simplest: Search a moving window and perform correlation

E.g. SSD (sum of squared differences)

Alternatively, SAD (sum of absolute

differences), CC (cross correlation), etc.

Usually restricted search range

Epipolar Geometry of a Stereo Pair

Example, SSD, win=5, search range=15

Correlation-Based Stereo Matching

[implementation by Rohit Singh and Mitul Saha]

Example, SAD, win=5, search range=8

ADCensus [Mei’11] (#1 Middlebury stereo benchmark, Sept. 2013)

A Modern Technique

Towards Multiple Views and Self-Calibration

-- The Structure-from-motion Pipeline --

Voodoo Camera Tracker - Steps of camera tracking

image sequence

Feature detection

correspondence

analysis

Outlier EliminationIncremental

Bundle AdjustmentSelf-Calibration

camera parameters

Correspondence Analysis

For video KLT-Tracker (Kanade, Lucas and Tomasi)

Minimization of the SSD between two windows in subsequent camera images to find the displacement vector

Correspondence Analysis – KLT Tracker

window

image K-1 image K

Linearization of the image signal with a Taylor series

KLT Tracker

Finding the minimum by setting the derivative to zero

KLT Tracker

Optical Flow – Horn & Schunck’81

Optical Flow as seen by a

person at the back of a train

• Continuous version of tracking: optical flow

• Apparent motion of brightness patterns in an image

sequence (typically two frames)

• For images: 𝑢( 𝑥): 𝑅2 𝑅2, is a vector valued fct.

• Often visualized as vector field or color coded

Example Yosemite sequencele

ftrigh

Flow field (middlebury coding)

Flow field (IPOL coding)

IPOL coding

middlebury coding

Assume a video 𝐼( 𝑥, 𝑡): 𝑅3 𝑅

Brightness constancy implies𝐼 𝑥 + 𝑢( 𝑥, 𝑡), 𝑡 = 𝐼( 𝑥, 𝑡 + 1)

Look at one particular time step with flow vectors 𝑢 = (𝑢𝑥, 𝑢𝑦)

perform Taylor expansion of 𝐼 𝑥, 𝑡 + 1 :

𝐼 𝑥, 𝑡 + 1 ≈ 𝐼 𝑥, 𝑡 +𝜕𝐼

𝜕𝑥𝑢𝑥 +

𝜕𝐼

𝜕𝑦𝑢𝑦 +

𝜕𝐼

𝜕𝑡+ 𝑂(𝛻2)

implies𝜕𝐼

𝜕𝑥𝑢𝑥 +

𝜕𝐼

𝜕𝑦𝑢𝑦 +

𝜕𝐼

𝜕𝑡= 0

Alternative form: 𝛻𝐼 ∙ 𝑢 +𝜕𝐼

𝜕𝑡= 0

Optical Flow- Derivation

KLT and optical flow descriptions are differential in nature only suitable for small displacements (up to 2 pixels)

Large displacements are small on a coarser scale

compute displacements on coarse scale

Upsample

Iteratively compute residuals on finer scales

KLT-Tracker: Resolution pyramids

resolution

KLT-Tracker: Affine mapping for longer sequences

For video (small displacements)

Minimization of the SSD between two windows in subsequent camera images

Linearization of the image signal with a Taylor series → linear equation system

Linearization only valid for small displacements → Resolution Pyramids

KLT-Tracker summary

Voodoo Camera Tracker - Steps of camera tracking

image sequence

Feature detection

correspondence

analysis

Outlier EliminationIncremental

Bundle AdjustmentSelf-Calibration

camera parameters

RANSAC (Random Sample Consensus) method

Outlier Elimination

RANSAC example – line fit

Least-squares fit

LS-fit after RANSAC

Fundamental Matrix (F-Matrix):

Outlier Elimination - Fundamental Matrix

Fundamental Matrix Estimation

multiple of these equations gives a linear equation system

Estimating the fundamental matrix from

feature correspondence:

2D Homography (H-Matrix):

Outlier Elimination - 2D Homography

2D Homography Matrix Estimation

multiple of these equations gives a linear equation system

Estimating the 2D homography matrix

from feature correspondence:

Outlier Elimination - Camera Matrix

Camera matrix (A-Matrix):

Incremental Bundle Adjustment

Bundle Adjustment

Measurement vector

Levenberg Marquardt –Non-linear Least Squares

Parameter vector

Taylor approximation:

with N x M Jacobian Matrix

transformed to linear least squares problem for each iteration

Linear least squares problem can be solved with normal equations

use to update solution iteratively

Levenberg Marquardt uses slightly different normal equations

Original normal equations

Modified normal equations

Lambda is changed during optimization

successful iteration

failed iteration

small ~ Newton style (quadratic convergence)

large ~ Gradient descent style (guaranteed decrease)

Requirements for Levenberg Marquardt minimization

Function to compute f

Start value X0

Optionally, function to compute J(but numerical derivation works as well)

Levenberg Marquardt

For bundle adjustment the problem becomes to large

(100 cameras + 10000 3D object points = 31200 parameters)

Can achieve huge speed-up by exploiting sparse structure of Jacobianmatrix

Partition parameters

partition A

partition B (only dependent on A and itself)

(typically A contains camera parameters, and B contains 3D points)

Sparse Levenberg Marquardt

Jacobian becomes

Normal equations

become

Corresponding block structure is

where denotes augmented by multiplying its diagonal entries by a factor of 1 + , and likewise. Left multiplication with

yields

which can be used to find with

which may be back-substituted to get with

Jacobian for bundle adjustment has sparse block structure

A1 A2 A3 P

12 x m 3 x n(in general much larger)

im.pts.

Needed for non-linear minimization

Voodoo camera tracker – demo session

Why self-calibration?

Allows flexible acquisition

No prior calibration necessary

Possibility to vary intrinsic camera parameters

Use archive footage

Self-calibration

We want to find a 4x4 transformation matrix that transforms all projective cameras into metric cameras

This does not change the back-projections onto the feature points

Self-calibration

Voodoo Camera Tracker –Some applications

Virtual advertising

Voodoo Camera Tracker –More applications

Car navigation

Architectural visualisation

3D Endoscopy

UAV terrain reconstruction

Matchmoving in Cloverfield

References

1. Multiple View Geometry in Computer Vision, Richard

Hartley and Andrew Zisserman, 2nd Edition

2. Triggs, B.: “Autocalibration and the absolute quadric”.

In “IEEE Conference on Computer Vision and Pattern

Recognition”, S. 609–614. 1997

3. Pollefeys, M., Koch, R., Gool, L. V.: “Self-Calibration

and Metric Reconstruction in Spite of Varying and

Unknown Internal Camera Parameters”. In “EEE

International Conference on Computer Vision, S. 90–

95. 1998

Computational Optical Imaging - Optique Numerique...

Documents

Information numerique

INCLUSION NUMERIQUE ET RAPPORT BIBLIOTHEQUES STAGE … Rapport Stage de professionnalisation Page 2 sur 68 ... ainsi pu accélérer le projet de généralisation de la fibre optique,

Cours d’electronique numerique - Freebertrand.granado.free.fr/Licence/ue201/coursimp.pdf · 2009-01-29 · Cours d’electronique numerique Cours d’electronique numerique Bertrand

Computational Optical Imaging - Optique Numerique ...giana.mmci.uni-saarland.de/.../lecture02_deconvolution.pdfLarge Linear Systems iterative solution methods require only matrix-vector

EFFETS DE LA TRANSITION NUMERIQUE TIC …...communication machine à machine…) ou de connectivité (large bande passante mobile, fibre optique…). Tantôt solution à part entière,

Identite Numerique

Analyse Numerique

Computational Optical Imaging - Optique Numeriquegiana.mmci.uni-saarland.de/website-template/... · Joseph Nicéphore Niépce 1765 - 1833 First photograph. Ivo Ihrke / Winter 2013

L’Identite Numerique

Bureau numerique Mobile et cartable numerique Milliweb

Optique Géométrique Optique Physique.pdf

Électromagnétisme Optique Physique Interférences Optique Ondulatoire L’optique ... · 2018. 1. 31. · Optique Ondulatoire Déﬁnition L’optique physique ou optique ondulatoire

système numerique

Culture numerique

Repertoire Numerique

Jounee numerique

Accessibilite Numerique

L’Enregistrement Optique : Comment ça marchesdg-master.com/lesestoff/2-GIS-DON-archivage-numerique...Jean-José Wanègue : « L’enregistremJean-José Wanègue : « Les disques

Électronique numerique

Formation Culture numerique - Focus sur le livre numerique -