8/10/2019 Photo Stitching
1/70
10/25/11
Image Stitching
Computational Photography
Derek Hoiem, University of Illinois
Photos by Russ Hewett
8/10/2019 Photo Stitching
2/70
Last Class: Keypoint Matching
K. Grauman, B. Leibe
Af Bf
A1
A2A3
Tffd BA ),(
1. Find a set of
distinctive key-points
3. Extract and
normalize the
region content
2. Define a region
around each
keypoint
4. Compute a localdescriptor from the
normalized region
5. Match local
descriptors
8/10/2019 Photo Stitching
3/70
Last Class: Summary
Keypoint detection: repeatable
and distinctive
Corners, blobs
Harris, DoG
Descriptors: robust and selective
SIFT: spatial histograms of gradient
orientation
8/10/2019 Photo Stitching
4/70
Today: Image Stitching
Combine two or more overlapping images to
make one larger image
Add example
Slide credit: Vaibhav Vaish
http://robots.stanford.edu/cs223b07/notes/CS223B-L11-Panoramas.ppthttp://robots.stanford.edu/cs223b07/notes/CS223B-L11-Panoramas.ppt8/10/2019 Photo Stitching
5/70
Problem basics
Do on board
8/10/2019 Photo Stitching
6/70
Basic problem
x = K [R t] X
x = K [R t] X
t=t=0
x=Hx where H = K R R-1K-1
Typically only R and f will change (4 parameters),
but, in general, H has 8 parameters
f f'
.x
x'
X
8/10/2019 Photo Stitching
7/70
Views from rotating camera
Camera Center
8/10/2019 Photo Stitching
8/70
Image Stitching Algorithm Overview
1. Detect keypoints
2. Match keypoints
3. Estimate homography with four matchedkeypoints (using RANSAC)
4. Project onto a surface and blend
8/10/2019 Photo Stitching
9/70
Image Stitching Algorithm Overview
1. Detect/extract keypoints (e.g., DoG/SIFT)
2. Match keypoints (most similar features,
compared to 2ndmost similar)
8/10/2019 Photo Stitching
10/70
Computing homography
Assume we have four matched points: How do
we compute homography H?
Direct Linear Transformation (DLT)
Hxx'
0h
vvvvuvu
uuvuuvu
1000
0001
'
''
''
'
w
vw
uw
x
987
654
321
hhh
hhhhhh
H
9
8
7
6
5
4
3
2
1
h
h
h
h
h
h
h
h
h
h
8/10/2019 Photo Stitching
11/70
Computing homography
Direct Linear Transform
Apply SVD: UDVT=A
h = Vsmallest(column of Vcorr. to smallest singular value)
987
654
321
9
2
1
hhh
hhh
hhh
h
h
h
Hh
0Ah0h
nnnnnnn vvvvuvu
vvvvuvu
uuvuuvu
1000
1000
0001
1111111
1111111
Matlab[U, S, V] = svd(A);
h = V(:, end);
8/10/2019 Photo Stitching
12/70
Computing homography
Assume we have four matched points: How do
we compute homography H?
Normalized DLT
1. Normalize coordinates for each imagea) Translate for zero mean
b) Scale so that u and v are ~=1 on average
This makes problem better behaved numerically (see
Hartley and Zisserman p. 107-108)2. Compute Husing DLT in normalized coordinates
3. Unnormalize:
Txx~ xTx ~
THTH ~1
ii Hxx
8/10/2019 Photo Stitching
13/70
Computing homography
Assume we have matched points with outliers:
How do we compute homography H?
Automatic Homography Estimation with RANSAC
8/10/2019 Photo Stitching
14/70
RANSAC: RANdom SAmple Consensus
Scenario: Weve got way more matched points
than needed to fit the parameters, but were notsure which are correct
RANSAC Algorithm
Repeat N times1. Randomly select a sample
Select just enough points to recover the parameters
2. Fit the model with random sample
3. See how many other points agree Best estimate is one with most agreement
can use agreeing points to refine estimate
8/10/2019 Photo Stitching
15/70
Computing homography
Assume we have matched points with outliers: How do
we compute homography H?Automatic Homography Estimation with RANSAC
1. Choose number of samples N
2. Choose 4 random potential matches
3. Compute Husing normalized DLT
4. Project points from xto x for each potentiallymatching pair:
5. Count points with projected distance < t
E.g., t = 3 pixels6. Repeat steps 2-5 Ntimes
Choose Hwith most inliers
HZ Tutorial 99
ii Hxx
http://users.cecs.anu.edu.au/~hartley/Papers/CVPR99-tutorial/tut_4up.pdfhttp://users.cecs.anu.edu.au/~hartley/Papers/CVPR99-tutorial/tut_4up.pdf8/10/2019 Photo Stitching
16/70
Automatic Image Stitching
1. Compute interest points on each image
2. Find candidate matches
3. Estimate homography H using matched pointsand RANSAC with normalized DLT
4. Project each image onto the same surface andblend
8/10/2019 Photo Stitching
17/70
Choosing a Projection Surface
Many to choose: planar, cylindrical, spherical,
cubic, etc.
8/10/2019 Photo Stitching
18/70
Planar Mapping
f f
x
x
1) For red image: pixels are already on the planar surface
2) For green image: map to first image plane
8/10/2019 Photo Stitching
19/70
Planar vs. Cylindrical Projection
Planar
Photos by Russ Hewett
8/10/2019 Photo Stitching
20/70
Planar vs. Cylindrical Projection
Planar
8/10/2019 Photo Stitching
21/70
Cylindrical Mapping
ff
x
x
1) For red image: compute h, theta on cylindrical surface from (u, v)
2) For green image: map to first image plane, than map to cylindrical surface
8/10/2019 Photo Stitching
22/70
Planar vs. Cylindrical Projection
Cylindrical
8/10/2019 Photo Stitching
23/70
Planar vs. Cylindrical Projection
Cylindrical
8/10/2019 Photo Stitching
24/70
Planar
Cylindrical
8/10/2019 Photo Stitching
25/70
Simple gain adjustment
8/10/2019 Photo Stitching
26/70
Automatically choosing images to stitch
8/10/2019 Photo Stitching
27/70
Recognizing Panoramas
Brown and Lowe 2003, 2007Some of following material from Brown and Lowe 2003 talk
8/10/2019 Photo Stitching
28/70
Recognizing Panoramas
Input: N images
1. Extract SIFT points, descriptors from all
images
2. Find K-nearest neighbors for each point (K=4)
3. For each image
a) Select M candidate matching images by counting
matched keypoints (M=6)
b) Solve homography Hijfor each matched image
8/10/2019 Photo Stitching
29/70
Recognizing Panoramas
Input: N images
1. Extract SIFT points, descriptors from all
images
2. Find K-nearest neighbors for each point (K=4)
3. For each image
a) Select M candidate matching images by counting
matched keypoints (M=6)
b) Solve homography Hijfor each matched image
c) Decide if match is valid (ni > 8 + 0.3 nf)
# inliers # keypoints inoverlapping area
8/10/2019 Photo Stitching
30/70
RANSAC for Homography
Initial Matched Points
8/10/2019 Photo Stitching
31/70
RANSAC for Homography
Final Matched Points
8/10/2019 Photo Stitching
32/70
Verification
8/10/2019 Photo Stitching
33/70
RANSAC for Homography
8/10/2019 Photo Stitching
34/70
Recognizing Panoramas (cont.)
(now we have matched pairs of images)
4. Find connected components
8/10/2019 Photo Stitching
35/70
Finding the panoramas
d h
8/10/2019 Photo Stitching
36/70
Finding the panoramas
d h
8/10/2019 Photo Stitching
37/70
Finding the panoramas
i i ( )
8/10/2019 Photo Stitching
38/70
Recognizing Panoramas (cont.)
(now we have matched pairs of images)
4. Find connected components
5. For each connected component
a) Perform bundle adjustment to solve for rotation
(1, 2, 3) and focal lengthf of all cameras
b) Project to a surface (plane, cylinder, or sphere)
c) Render with multiband blending
B dl dj f i hi
8/10/2019 Photo Stitching
39/70
Bundle adjustment for stitching
Non-linear minimization of re-projection error
where H= K R R-1K-1
Solve non-linear least squares (Levenberg-
Marquardt algorithm) See paper for details
),(1
N M
j k
i
disterror xx
Hxx
B dl Adj
8/10/2019 Photo Stitching
40/70
Bundle Adjustment
New images initialized with rotation, focal
length of the best matching image
B dl Adj
8/10/2019 Photo Stitching
41/70
Bundle Adjustment
New images initialized with rotation, focal
length of the best matching image
Bl di
8/10/2019 Photo Stitching
42/70
Blending
Gain compensation: minimize intensity
difference of overlapping pixels
Blending
Pixels near center of image get more weight
Multiband blending to prevent blurring
M lti b d Bl di (L l i P id)
8/10/2019 Photo Stitching
43/70
Multi-band Blending (Laplacian Pyramid)
Burt & Adelson 1983
Blend frequency bands over rangel
M ltib d bl di
8/10/2019 Photo Stitching
44/70
Multiband blending
Bl di i (IJCV 2007)
8/10/2019 Photo Stitching
45/70
Blending comparison (IJCV 2007)
Bl di C i
8/10/2019 Photo Stitching
46/70
Blending Comparison
St i ht i
8/10/2019 Photo Stitching
47/70
Straightening
Rectify images so that up is vertical
F rther reading
8/10/2019 Photo Stitching
48/70
Further reading
Harley and Zisserman: Multi-view Geometry book
DLT algorithm: HZ p. 91 (alg 4.2), p. 585 Normalization: HZ p. 107-109 (alg 4.2)
RANSAC: HZ Sec 4.7, p. 123, alg 4.6
Tutorial:
http://users.cecs.anu.edu.au/~hartley/Papers/CVPR99-tutorial/tut_4up.pdf
Recognising Panoramas: Brown and Lowe, IJCV2007 (also bundle adjustment)
Tips and Photos from Russ Hewett
http://users.cecs.anu.edu.au/~hartley/Papers/CVPR99-tutorial/tut_4up.pdfhttp://users.cecs.anu.edu.au/~hartley/Papers/CVPR99-tutorial/tut_4up.pdfhttp://cvlab.epfl.ch/~brown/papers/ijcv2007.pdfhttp://cvlab.epfl.ch/~brown/papers/ijcv2007.pdfhttp://cvlab.epfl.ch/~brown/papers/ijcv2007.pdfhttp://cvlab.epfl.ch/~brown/papers/ijcv2007.pdfhttp://users.cecs.anu.edu.au/~hartley/Papers/CVPR99-tutorial/tut_4up.pdfhttp://users.cecs.anu.edu.au/~hartley/Papers/CVPR99-tutorial/tut_4up.pdfhttp://users.cecs.anu.edu.au/~hartley/Papers/CVPR99-tutorial/tut_4up.pdf8/10/2019 Photo Stitching
49/70
Tips and Photos from Russ Hewett
Capturing Panoramic Images
8/10/2019 Photo Stitching
50/70
Capturing Panoramic Images
Tripod vs Handheld
Help from modern cameras
Leveling tripodGigapan
Or wing it
Exposure
Consistent exposure between frames
Gives smooth transitions
Manual exposure
Makes consistent exposure of dynamic scenes easierBut scenes dont have constant intensity everywhere
Caution
Distortion in lens (Pin Cushion, Barrel, and Fisheye)
Polarizing filters
Sharpness in image edge / overlap region
Image Sequence
Requires a reasonable amount of overlap (at least 15-30%)
Enough to overcome lens distortion
Pikes Peak Highway CO
8/10/2019 Photo Stitching
51/70
Photo: Russell J. Hewett
Pike s Peak Highway, CO
Nikon D70s, Tokina 12-24mm @ 16mm, f/22, 1/40s
Pikes Peak Highway CO
8/10/2019 Photo Stitching
52/70
Photo: Russell J. Hewett
Pike s Peak Highway, CO
(See Photo On Web)
360 Degrees Tripod Leveled
http://photos.russellhewett.com/Colorado/Colorado-Panorama/2694311_sUXTbhttp://photos.russellhewett.com/Colorado/Colorado-Panorama/2694311_sUXTb8/10/2019 Photo Stitching
53/70
Photo: Russell J. Hewett
360 Degrees, Tripod Leveled
Nikon D70, Tokina 12-24mm @ 12mm, f/8, 1/125s
8/10/2019 Photo Stitching
54/70
Handheld Camera
8/10/2019 Photo Stitching
55/70
Photo: Russell J. Hewett
Handheld Camera
Nikon D70s, Nikon 18-70mm @ 70mm, f/6.3, 1/200s
Handheld Camera
8/10/2019 Photo Stitching
56/70
Photo: Russell J. Hewett
Handheld Camera
Les Diablerets, Switzerland
8/10/2019 Photo Stitching
57/70
Photo: Russell J. Hewett
Les Diablerets, Switzerland
(See Photo On Web)
Macro
http://photos.russellhewett.com/Switzerland/Switzerland-Preview/13751686_fpNfKhttp://photos.russellhewett.com/Switzerland/Switzerland-Preview/13751686_fpNfK8/10/2019 Photo Stitching
58/70
Photo: Russell J. Hewett & Bowen Lee
Macro
Nikon D70s, Tamron 90mm Micro @ 90mm, f/10, 15s
Side of Laptop
8/10/2019 Photo Stitching
59/70
Photo: Russell J. Hewett & Bowen Lee
Side of Laptop
Considerations For Stitching
8/10/2019 Photo Stitching
60/70
Co s de at o s o St tc g
Variable intensity across the total scene
Variable intensity and contrast between frames
Lens distortion
Pin Cushion, Barrel, and Fisheye
Profile your lens at the chosen focal length (read from EXIF)
Or get a profile from LensFun
Dynamics/Motion in the scene
Causes ghosting
Once images are aligned, simply choose from one or the other
Misalignment
Also causes ghostingPick better control points
Visually pleasing result
Super wide panoramas are not always pleasant to look at
Crop to golden ratio, 10:3, or something else visually pleasing
Ghosting and Variable Intensity
8/10/2019 Photo Stitching
61/70
Photo: Russell J. Hewett
g y
Nikon D70s, Tokina 12-24mm @ 12mm, f/8, 1/400s
8/10/2019 Photo Stitching
62/70
Photo: Russell J. Hewett
Ghosting From Motion
8/10/2019 Photo Stitching
63/70
Photo: Bowen Lee
g
Nikon e4100 P&S
Motion Between Frames
8/10/2019 Photo Stitching
64/70
Photo: Russell J. Hewett Nikon D70, Nikon 70-210mm @ 135mm, f/11, 1/320s
8/10/2019 Photo Stitching
65/70
Photo: Russell J. Hewett
Gibson City, IL
8/10/2019 Photo Stitching
66/70
Photo: Russell J. Hewett
y
(See Photo On Web)
Mount Blanca, CO
http://photos.russellhewett.com/Random-and-Misc/Contests/3889511_cGhzmhttp://photos.russellhewett.com/Random-and-Misc/Contests/3889511_cGhzm8/10/2019 Photo Stitching
67/70
Photo: Russell J. Hewett Nikon D70s, Tokina 12-24mm @ 12mm, f/22, 1/50s
Mount Blanca, CO
8/10/2019 Photo Stitching
68/70
Photo: Russell J. Hewett (See Photo On Web)
Things to remember
http://photos.russellhewett.com/Colorado/Colorado-Previews/2605757_evd3vhttp://photos.russellhewett.com/Colorado/Colorado-Previews/2605757_evd3v8/10/2019 Photo Stitching
69/70
Things to remember
Homography relates rotating cameras
Recover homography using RANSAC andnormalized DLT
Can choose surface of projection: cylinder,plane, and sphere are most common
Lots of room for tweaking (blending,straightening, etc.)
Next class
8/10/2019 Photo Stitching
70/70
Next class
Using interest points to find objects indatasets