Upload
kay-robinson
View
41
Download
1
Tags:
Embed Size (px)
DESCRIPTION
Generating Seamless Stereo Mosaics from Aerial Video. Zhigang Zhu Allen R. Hanson, Harpal S. Bassali Howard Schultz, Edward M. Riseman Computer Vision Lab Computer Science Department University of Massachusetts at Amherst [email protected] http://www.cs.umass.edu/~zhu. Introduction. - PowerPoint PPT Presentation
Citation preview
04/19/23 1
Generating Seamless Stereo
Mosaics from Aerial Video
Zhigang ZhuAllen R. Hanson, Harpal S. Bassali
Howard Schultz, Edward M. Riseman
Computer Vision Lab
Computer Science Department
University of Massachusetts at Amherst
http://www.cs.umass.edu/~zhu
04/19/23 2
Introduction
ObjectivesDevelop methods to automatically generate geo-referenced stereo mosaics from video sequences
DefinitionsFree Mosaic: composite of video sequence by registering overlapping frames
• subject to drift relative to the terrain• constrains: pure rotation or planar scenes
Geo-Mosaic: use 3D instrumentation to constrain mosaic to world coordinates Stereo Mosaics : a pair of mosaics from a single camera
• seamless under motion parallax• preserve 3D information• can be viewed in 3D directly
04/19/23 3
Important Issues in 3D Video Mosaicing
Representationcompact representation for a large-scale 3D sceneorthogonal (DEM), perspective (mosaic), parallel-perspective
Computationhow expensive are the computations of the algorithms?Goal: affordable, efficient and robust mosaicing
Accuracyaccurate for 3D viewing and 3D reconstruction ?
04/19/23 4
Geometry, Representation and Properties
Sensor motion is pure translation
Sensor
Image Plane
“Right” Mosaic “Left” Mosaic
04/19/23 5
Re-Organizing the images….
04/19/23 6
Recovering Depth from Mosaics
parallel-perspective stereo mosaics
Depth accuracy independent of depth
Two views from different perspective stereo
P(X,Y,Z)
Height H from Laser Profiler
GPS
)(y
y
y
y
d
ydH
d
BFZ
disparity
baseline
Fixed !
displacement
04/19/23 7
Stereo mosaics of Amazon rain forest
166-frame telephoto video sequence -> 7056*944 mosaics
a
b
c
04/19/23 8
Stereo viewing
Red: Right view; Blue/Green: Left view
04/19/23 9
Computation: how expensive in in real world
application?
Pro-processingArbitrary motion other than a 1D translation / 3D translationCamera orientation estimation and image rectification
Mosaicing Image sequence is not dense enough for seamless mosaicshow to generate parallel-perspective projection ?
3D recoveryhow expensive is the match in stereo mosaics?Baseline and epipolar geometry of stereo mosaics
Post-processingparallel-perspective 3D mosaics to an orthogonal DEM
04/19/23 10
Step 1. Pro-ProcessingMotion estimation & rectification
Camera pose estimation using navigation instrumentation and bundle adjustment
only sparse tie points widely distributed in the two images are needed
Image rectificationtransformation on two narrow slices in each frame
Dominant motion direction
3D path of the camera: 3D rotation + 3D translation
Original image frames
Dominant motion direction Y
Rectified image frames
Path of the camera: 3D translation
04/19/23 11
Step 2. 3D Mosaicingseamless mosaicing with motion parallax
Front slit : Both slits are sub-images of m columns (m>=1)Rear slit
Perspective image
Multi-perspective mosaics
Parallel-perspective mosaics
Rays of left view
Left view mosaic
……
Rays of right view
Right view mosaic
……
Rays of right view
Right view mosaic
……
Rays of left view
Left view mosaic
……
View interpolationRay interpolation
(1)
(2)
In a multi-perspective projection mosaic, each sub-image is full perspective, but sub-images from different frames will have different viewpoints. This may cause seams in the mosaic due to motion parallax.
Geometric Seams -Clearly Visible to Human Eyes, especially along depth boundaries
Introduce Error in Height Estimation
04/19/23 12
PRISM parallel ray interpolation for stereo mosaicing
)(
)2/(
21
1
yyiy
xxxi
yy
yyi
tts
stt
syy
dytt
2
)2
( 11
yyii
y
y
xxii
dty
dy
s
sxtx
Interpolated view:
Mosaic coordinates:
- Take a slice of certain width from each frame- Perform local registration between the overlapping slices- Generate parallel interpolated views between two known views- Re-project the point back to the mosaic
(Tx, Ty)
IP of 1st fixed line
1st fixed line
y0= dy/2
(Tx+Sx, Ty+Sy)
IP of 2nd fixed line
2nd fixed line
(X,Y,Z)
(x1, y1)
(x2, y2)
IP of interpolated fixed line
(Txi,Tyi)
(xi, yi)
04/19/23 13
Comparison : 2D mosaic & 3D mosaic
2D mosaic from dense image sequence ( 4-pixel interframe motion)2D mosaic from sparse image sequence ( 40-pixel interframe motion)
3D mosaic from sparse image sequence ( 40-pixel interframe motion)
04/19/23 14
Step 3. 3D reconstructionepipolar geometry of stereo mosaics
epipolar curveylr
lrrlxlr dyy
yyyybxx
),(
Epipolar curve1D search
Near horizontal line
Coverged pairsmall search region
xl
yl
left mosaic
xr
yr
right mosaic
dy/2
(xl,yl)
(xl,yl
)
txl(yl)
(xr,yr
)txr(yr)- txl(yl)dy/2
txr(yr)
04/19/23 15
Epipolar Geometry in Real Stereo mosaics
“Left” Mosaic
04/19/23 16
Epipolar Geometry in Real Stereo mosaics
“Right” Mosaic
04/19/23 17
Epipolar Geometry in Real Stereo mosaics
Depth Map: Brighter is higher elevation
04/19/23 18
Step 4. Post-Processing from 3D mosaic to DEM
Just a transformation !!
(X,Y,Z) world coordinates(xl,yl) left mosaic coordinates(xr,yr) right mosaic coordinates
H : a reference height (depth)F: focal length of the camerady: distance between left and right slitsby: adaptive baseliney = yr - yl
2 ,)1( lr
ly
yy
F
HYx
d
y
F
HX
)1(
yy
y
d
yH
d
bHZ
04/19/23 19
3D Rendering Result
04/19/23 20
Motion Refinement for Geo-Mosaic- when geo-data is not accurate
Extended Kalman Filter (EKF) approach
Unconstrained
-Image Match
-Accumulating error
Ground TruthGeo Meaursement
- absolute error EKF EsimationFlying path
Time Update
“Predict” from
image registration
Measurement Update
“correct”
by geo-data Motion parameter
A: Warp matrix t : translation
04/19/23 21
Geo-Reference Mosaicsframe-by frame mosaicing
Geo-mosaic
Free Mosaic
04/19/23 22
Geo-Reference Mosaicsglobal warping from free mosaic
Free mosaic
Geo-mosaic from free mosaic
square
straight
Frame-by-frame geo-mosaic
diamond
curve
04/19/23 23
Methods Summarydistribute the computations in four steps
Pro-processingMotion estimation: sparse tie points distirbuted in entire frames
Rectification and Mosaicing (PRISM)
Process two narrow slices
3D recovery (Terrest)stereo match only in two mosaics
Post-processingjust a coordinate transformation
04/19/23 24
Accuracy of 3D from Stereo Mosaics
Adaptive baselines and fixed disparity - uniform depth resolution
Ray interpolation between two successive views is similar to image rectification
3D recovery accuracy is comparable to that of a perspective stereo with an optimal baseline
04/19/23 25
Next Steps
Camera calibration and bundle adjustmentsGeometric and Photometric SeamlessnessUsing Structure Information and video –photo matchingError analysis in geo-referenced mosaic and 3D reconstruction using ground truth