Geometric Transformations(based on Joan Serrat slides)

Felipe LumbrerasDept. Ciències de la Computació / Centre de Visió per Computador

Universitat Autònoma de Barcelonahttp://www.cvc.uab.es/shared/teach/a102784/

Geometric Transformations

• Introduction

• Mapping computation– Scene Sensor modelling

– Control points

– Semi-automatic control point determination

– Automatic control point determination

• Resampling

• Applications– Registration

– Bird’s view

– Mosaic of images

Introduction

Image transforms according to their formulation :

• point

• local neighbourhood

Introduction

Image transforms according to their formulation :

• global

• arithmetic (+,-,*,/)

u

v

Introduction

Image transforms according to their formulation :

• geometric

transform coordinates, not pixel values

Introduction

Main applications:

• Correct image distortions caused by acquisition devices or geometry

• Compare images

Fields:

• Remote sensing

• Medical imaging

• …

Introduction

Camera distortions

Introduction

Satellite images

Introduction

Satellite images

Orthorectified image:

each pixel “seen

from above”

Introduction

Satellite images

Sensor fusion

IKONOS4m color

1m panchromatic fusion(pansharpening)

+ =

Introduction

Medical Imaging

Frame registration on eye angiographies

original registered

Introduction

Medical Imaging

Frame registration on eye angiographies

original registered

Introduction

Medical Imaging

Frame registration on eye angiographies

original registered

Introduction

A geometric transform has two steps:

1. Computation of the mappingFind the equation relating the coordinates systems of original and result images.

2. Resampling or interpolationAssign values to the pixels of the result image from those of the input image.

Mapping

Introduction (Output to input)

Input to output :problem?

Output to input

Introduction

Zoom

input to output output to input

input

outputs

Introduction

Rotation

input to output output to input

input

outputs

Mapping computation

• Model mathematically the acquisition process and invert it– Example: earth rotation,

effect in satellite images

– Example: rolling shutter

in photogrammetry

𝑥 = 𝑘 𝑅 −𝑅𝑐 𝑋 𝑥 = 𝑘 𝑅 −𝑅(𝑡)𝑐(𝑡) 𝑋rolling shutter

modelglobal shutter

model

Mapping computation

• Adjustment to control points

Mapping computation

• Adjustment to control points

input data: pairs of corresponding control points

𝑘 = 𝑞1 + 𝑞2𝑖 + 𝑞3𝑗 + 𝑞4𝑖2 + 𝑞5𝑖𝑗 + 𝑞6𝑗

2 +⋯

𝑙 = 𝑟1 + 𝑟2𝑖 + 𝑟3𝑗 + 𝑟4𝑖2 + 𝑟5𝑖𝑗 + 𝑟6𝑗

2 +⋯

𝑖𝑚, 𝑗𝑚 , 𝑘𝑚, 𝑙𝑚 , 𝑚 = 1…𝑛

Mapping computation• Adjustment to control points

• Matrix notation

𝑄 𝑖, 𝑗 = 𝑘 = 𝑞1 + 𝑞2𝑖 + 𝑞3𝑗 + 𝑞4𝑖2 + 𝑞5𝑖𝑗 + 𝑞6𝑗

2 +⋯

𝑅 𝑖, 𝑗 = 𝑙 = 𝑟1 + 𝑟2𝑖 + 𝑟3𝑗 + 𝑟4𝑖2 + 𝑟5𝑖𝑗 + 𝑟6𝑗

2 +⋯

min𝑟1…𝑟𝑝

𝑚=1

𝑛

𝑙𝑚 − 𝑅 𝑖𝑚, 𝑗𝑚2

min𝑞1…𝑞𝑝

𝑚=1

𝑛

𝑘𝑚 − 𝑄 𝑖𝑚, 𝑗𝑚2

𝐾 = [𝑘1, 𝑘2, … 𝑘𝑛]𝑇, 𝐿 = [𝑙1, 𝑙2, … 𝑙𝑛]

𝑇 , 𝑄 = [𝑞1, 𝑞2, … 𝑞𝑛]𝑇, 𝑅 = [𝑟1, 𝑟2, … 𝑟𝑛]

𝑇

M=

1 𝑖1 𝑗1 𝑖12 𝑖1𝑗1 𝑗1

2 𝑖13 ⋯

1 𝑖2 𝑗2 𝑖22 𝑖2𝑗2 𝑗2

2 𝑖23 ⋯

⋮⋮

1 𝑖𝑛 𝑗𝑛 𝑖𝑛2 𝑖𝑛𝑗𝑛 𝑗𝑛

2 𝑖𝑛3 ⋯

𝐾 = M𝑄 𝐿 = M𝑅

𝑄 = M𝑇M −1M𝑇𝐾

𝑅 = M𝑇M −1M𝑇𝐿

Solution using pseudoinverse matrix

Mapping computation

• Adjustment to control points

Manually selecting a number of corresponding control points is:

– time consuming

– tedious

– error prone

– imprecise

Mapping computation

• Adjustment to control points

Semi-automatically: given control points on the input image, find the corresponding points on the reference (output)

Similarity measure: correlation, covariance, normalized cross-correlation

Mapping computation

• Adjustment to control points

Automatically: find a set of local features in both images and look for a transformation in a robust way (RANSAC: RANdom SAmple Consensus)

RANSAC

• RANdom SAmple Consensus: iterative method to estimate parameters of a mathematical model from a set of observed data which contains outliers.

1.- Select randomly the minimum number of points required to determine the model parameters. Solve for the parameters of the model.

2.- Determine how many points from the set of all points fit with a predefined tolerance (inliers).

3.- If inliers exceeds a predefined threshold τ, re-estimate the model parameters using all the identified inliers and terminate.

4.- Else repeat steps 1 through 3 a maximum of N times.

source: http://en.wikipedia.org/wiki/RANSAC

Resampling

Resampling

Given a certain mapping, fill in the output image by interpolating the pixel values of the input image at non integer coordinates

Resampling

• Zero order interpolation: nearest neighbour

Resampling

• First order interpolation: in 2D images bilinear

Resampling

• Third order interpolation: in 2D images bicubic

source: https://en.wikipedia.org/wiki/Bicubic_interpolation

Resampling

• Ideal interpolation for bandlimited signal sampled at Nyquist rate: convolution with sinc 𝑥, 𝑦

sinc 𝑥 =sin(𝑥)

𝑥

Nyquist rate: is the minimum rate at which a finite bandwidth (bandlimited) signal needs to be sampled to retain all of the information.

For a bandwidth of span B, Nyquist frequency = 2 B. If a signal is sampled at regular intervals dt, Nyquist rate = 1/(2dt)

sinc 𝑥, 𝑦

Resampling

• Comparison example180o rotation by reflection (no interpolation needed) versus 36 successive rotations of 5o each

. . .

5° 10° 15°

170° 175° 180°

original

Resampling

• Comparison example3 input images

Resampling

• Biliniar interpolation (2 x 2 pixels)

• Bicubic (4 x 4 pixels)

Resampling

• Lanczos kernel (6 x 6 pixels)

• sinc (x,y) (16 x 16 pixels)

Applications

Applications

• Registration

• Geometric alignment of images in order to compare or fusion pixel

by pixel.

• More used in medical imaging and remote sensing.

• Detects temporal differences or combine different sensors.

• Projections:

similarities, affinities, projective, spherical, cylindrical, warpings

Applications

• Registration : Geometric alignment for 3D fusion CT-MRI

CT MRI

Applications

• Registration : Geometric alignment

Sagital slicesAxial slices

MRI volume

Applications

• Registration :

Geometric alignment

Applications

• Registration : Alignment to stabilize a video sequence

Original sequence Stabilized sequence

Applications

• Registration :

Alignment to stabilize a

video sequence

A arteryV vein

Applications

• Construction of stitching mosaics from individual image

– Mathematical model of a camera: pin-hole, central projection.

– What a homography is?

– Mosaics of stitching images.

Applications

• Mathematical model of a camera: pin-hole, central

projection.

Applications

• Mathematical model of a camera: pin-hole, central

projection.

focal distance

holeimage

plane

Applications

• Mathematical model of a camera: pin-hole, central

projection.

focal distance

variation

Applications

• Mathematical model of a camera: pin-hole, central

projection.

photographic paper (negative)

1 2 3 4

Applications

• Mathematical model of a camera: pin-hole, central projection.

Applications

• What a homography is?

A particular case: central projection of a plane into a another plane

(2D to 2D).

There are other projections: orthographic, weak-perspective

Applications

• What a homography is?

Examples of interest: changing the point of view in the image of a plane.

Applications

• What a homography is?

Examples of interest: changing the point of view

in the image of a plane.

Applications

• Panoramic mosaics

Applications

projecció rectilínia (homografia) 100°

Panoramics of Georges Lagarde (GURL)

Applications

projecció rectilínia (homografia) 130°

Panoramics of Georges Lagarde (GURL)

Applications

projecció rectilínia (homografia) 170°

Panoramics of Georges Lagarde (GURL)

Applications

Panoramics of Georges Lagarde (GURL)

Applications

Panoramics of Georges Lagarde (GURL)

Impossible ?

Applications

Panoramics of Georges Lagarde (GURL)

Applications

• Lane road detection

Applications

• Lane road detection

Applications

• Lane road detection

Applications

• Lane road detection

Applications

• Lane road detection

s

−60°

+60°

Radon transform

θ

s

θi

g(s,θi)

Applications

• Lane road detection

Applications

• Distances in a plane

Applications

• Distances in a plane

Mapas obtenido mediante imágenes aéreasTFG: Rubén Pérez Conte (2014)