An Autonomous 3-D Photogrammetric Approach to …vision.eecs.ucf.edu/workshop/sblask_reg.pdf · An Autonomous 3-D Photogrammetric Approach Steven G. Blask, ... MP7 + MP8 MP9 + + MP10

next level solutions13 July 2001Harris Proprietary

An Autonomous 3An Autonomous 3--D Photogrammetric ApproachD Photogrammetric Approach

Steven G. Blask, John A. Van Workum{sblask, jvanwork}@harris.com

Harris Corporation GCSD

to Airborne Video Geoto Airborne Video Geo--RegistrationRegistration


Outline

• Overview of Harris Registration Approach

• Airborne Video Extensions - PVR System

• DARPA AVS-PVR Processing Results

• Discussion


Impact of ESD Errors

Geo-Reference Image Video Frame


MP1

+

MP2

+

MP3

+

MP4

+

MP5

+

MP6

+

MP7

+

MP8

+MP9

+

MP10MP10

+

MP11

+

MP12

+

MP13MP13

+

MP14

+

MP15

+

MP16

+

Photogrammetric Model Based Registration Overview

MP1

+

MP2

+

MP3

+

MP4

+

MP5

+

MP6

+

MP7

+

MP8

+MP9

+

MP10MP10

+

MP11

+

MP12

+

MP13MP13

+

MP14

+

MP15

+

MP16

+

AmbiguityResolution

ConvergedSolution

Sensor ModelAdjustment

DEM orSite Model

Initial Transformation Process• Image -> 3-D Surface -> 2D View• Tile Overlap Area in Scene Space• Build Common Neighborhood Windows

Normalized X-Correlation• In Enhanced Edge Space• Build Correlation Surface• Evaluate Correlation Peaks

Consistent Subset Analysis• Compute Mean Offset Vector• Sequentially Sort Peaks• Resolve Ambiguities & Outliers• Adjust Sensor Parameters• Repeat at Next Resolution Level

OutputAdjusted

Support Data

Support Data InitializesSensor Geometry Model

Registered Images

x, y, z, v, g.

x, y, z, v, g.

PeaksMatchPoints

AdjustedParameters


Initial Transformation Process

• The images are subsampled to create reduced resolution data sets• Software resampler creates patches at any required GSD on demand

Level5..

2

1

0

256

128

64

256

128

64



Sensor 1 Sensor 2


• Use a priori knowledge of each sensor imaging event and a Digital Elevation Model (DEM) to project imagery to the 3D terrestrial surface


Collected ImageCollected Image

Orthorectified Image

DEM

u = Pu (ϕ, λ, h)v = Pv (ϕ, λ, h)

GeometryModel forSensor 1

GeometryModel forSensor 2

(ϕ, λ, h) = F (line, sample,Xv, Yv, Zv, roll, pitch, heading,azimuth, elevation, twist,focal length)



• Orthorectification places the images in a common orientation with minimal distortion present (unmodelled buildings & trees still layover)


Normalized X-Correlation

Image 1Neighborhood

Pre-EmphasisFilter

Cross-Correlation

Image 2Neighborhood

3-D Cross-Correlation

Surface

Pre-EmphasisFilter

xij

3N = # of elements in reference patch

2N = # of elements in comparison patch

= elements in comparison patch

ijy = elements in reference patch

( )

��

�

�

��

�

�

��

��

�

−��

��

�

−

−=

��

��

��

= =

= =

= =

= =

= =

3

2

1 1

1 1

2

2

2

1 1

1 1

2

1 1

2

3

N

yy

N

xx

yxx

NNr

n

i

n

jijn

i

n

jij

n

i

n

jijn

i

n

jij

n

i

n

jijij

Correlation Patch

up to 4 candidates are chosen

ReferencePatch

ComparisonPatch


Matchpoint Display

Geo-Reference ImageryVideo Mission Image


+ + +

+ + + +

+ + + +

+ + + +

+

Estimate of Misalignment

• Multiple correlation peaks are computed for each grid point neighborhood

• A parametric hill finder is used to evaluate each peak

• The mean and standard deviation of registration error are calculated from the offset and average ellipse

• The best consistent subset of correlation peaks is chosen by sequential sorting

• Offset vectors imply global ground “correction” needed to improve registration, wild pt. editing eliminates outliers


Sensor Adjustment Process

• Sensor parameters are adjusted to minimize the error between ground projections of common match points

• Conjugate Gradient Search, Least Squares, and Kalman Filter adjustment algorithms

Image 2 geometry parameters

Minimal Perturbation Adjustment Procedure

Image 1 geometry parameters

Registration measurements

(correspondences)

x1, y1, z1, v1, v1.

x2, y2, z2, v2, v2.


Output & Derivative Products

• Improved telemetry used by Geolocation & Mosaic– Telemetry parameters initialize sensor model to define a 3D ray

through any pixel in the image, which may be intersected with the DEM to produce a geolocation or orthorectify a video frame.

– By improving telemetry, we improve geodetic accuracy of pixels.

N38.12.00, W077.24.00

N38.07.00, W077.18.45

N38.12.00, W077.18.45

N38.07.00, W077.24.00


Registration Solution

��

�

�

��

�

�

⋅++⋅+⋅= ��

��

�

∂∂

��

��

�

∂∂

��

��

�

∂∂

2

2

22

2

21

2

21n

ne

xf

xf

xf σσσ

�

��

�

�

��

�

�

⋅++⋅+⋅= ��

��

�

∂∂

��

��

�

∂∂

��

��

�

∂∂

2

2

22

2

21

2

21n

nxf

xf

xf σσσ

�

1x1x , 2x2x ,� nxnx parameterstherepresents parameterstherepresents

σσ σσ11 , 22 ,�

nσnσ ixofvariancestherepresents ixofvariancestherepresents

f parameterstheoffunction therepresents parameterstheoffunction therepresents

• Advantage of model-based approach: can perform rigorous error propagation to characterize geopositioning solutions and provide a posteriori error covariances for adjusted sensor model params

initial

registered

higher accuracy,tighter error bounds

RegisteredSensor 1 &Sensor 2


Airborne Video Extensions

Precision Video Registration SystemPrecision Video Registration System


PVR Architecture

Video andTelemetry

Video A/Dand

TelemetryDecoder

Video A/Dand

TelemetryDecoder

Reference& Frame

Repository

Reference& Frame

Repository

PrecisionGeolocationProcesses

PrecisionGeolocationProcesses

Telemetry &RegistrationProcesses

(RTVR)

Telemetry &RegistrationProcesses

(RTVR)

PrecisionMosaic

Processes

PrecisionMosaic

Processes

Video Video Frames

AdjustedTelemetry

RawTelemetry

Video Frames

C2 UserC2 User

IA UserIA User

Auto’dProc.

Auto’dProc.

PVRManager

PVRManager

� Controls PVR Processes� Handles Status Messages� Reports Metrics

Precision Broadcast

CAGS Services PVR ClientsPVR Processing


RTVR Architecture

Process Control

Telemetry Adjustments

Telemetry Database

RawTelemetry

Selected ESD

SelectedVideo Frames

Match PointSequential State

Estimator

MatchPointsLSE Worm

Controller

Observation CombinerPrecision

MatchPoints

MatchPoints

MatchPoints

PrecisionMatchPoints

PrecisionMatchPoints

MatchPoints

Observation Generators


Telemetry Queue/Database

Time

VideoFrames

30 Hz

Position

10 Hz

Attitude

30 Hz

Gimbals

25 Hz

Sensor

6.25 Hz

Raw Telemetry(from CAGS Metadata)

Telemetry Adjustment(from Match Pt. SSE)

HARD REGISTRATION

treq

tprev

tnextHARD REGISTRATION

SOFT REGISTRATION

Quality of Service:PrecisionExtrapolatedInterpolated

Quality of Service:PrecisionExtrapolatedInterpolatedPosition, Attitude,

Gimbals, Sensor DeltasGoal: 1 Hz

Oct’99 FEX: 0.22 Hz


Affine Consistent Subset

• Required to account for scale and rotation distortion


200 Unregistered Frames

Selected frames from (in order) 1 hour, sparsefeatures, class1, and class2 data sets, 29Mar99.

Raw telemetry errors in ascending order.


Original CSS Results


Registration errors for original consistentsubset criterion in ascending order.

Outliers due to fuselage obscuration and low elevation angles


Affine CSS Results


Registration errors for affine consistentsubset criterion in ascending order.

Outliers due to fuselage obscuration and extremely low low elevation angles (17-20 deg)


KF Adjustment Algorithm

• Sparse scene content of Airborne video requires accumulation of match points over space and time

• Kalman filter adjustment vs. N-frame co-registration– Adds one image at a time to solution– Only need to estimate parameters for one image– Smaller set of equations– No waiting for additional images

• State vector X models adjustments to telemetry; slowly varying bias suggests constant state model is suitable:


KF vs. Single Frame Results

Oct'99 VA D2 Clip

0

5

10

15

20

25

30

35

40

63 frames, EO vs DOQ DTED1

grou

nd e

rror

(m)

beforeautoregkftype

kf: median: 3.80 90th: 8.44 mean: 4.70 std dev: 2.50 <10m: 93.3%autoreg: median: 7.44 90th: 13.33 mean: 8.27 std dev: 3.62 <10m: 71.6%

before: median: 23.25 90th: 28.43 mean: 23.24 std dev: 4.08 <10m: 0.0%


KF vs. Single Frame Results

Oct'99 VA D2 Order Stats

0

5

10

15

20

25

30

35

63 frames, EO vs DOQ DTED1

grou

nd e

rror

(m)

beforeautoregkf


Scene Content & Prescreener

• Compute MPt normalized image space residuals:

• Apply thresholds– min. no. match points

• 9 for frame-to-mono ref.• 5 for frame-to-stereo ref.• 4 for frame-to-frame

– avg. norm. res. ≤≤≤≤ 1 pixel– max. norm. res. ≤≤≤≤ 2 pixels

��

��

�

−−

=

=

22

11

-1T

xyxy

Σρ

ε

εε

Correlation surfaceexamples

Scene contentexamples

StrongPeak

Ridges

WeakPeaks


Significant Scene Content Changes Low Salient Feature Density

Dynamic Video Worm

Successful Mission-to-Reference Single Frame Event

Prescreened Mission-to-Reference Single Frame Event,Successful Mission-to-Mission Match Points

Worm Anchor Mission-to-Reference Frame

Prescreened Mission-to-Mission, Unanchored End of Worm

Unanchored Worm Segment, Interpolated “Soft” Adjustment


DTED Accuracy

DTED Terrain Surface

Solution Point

Solution Covariance

Ray Covariance

Image Ray/Range Arc

Image 1

Video 1

Video 2

Geo-Ref

Further Accuracy Improvement

Geometry effects


PVR Georegistration Performance Using DOQ & DTED

Dynamic WormDynamic Worm(LSE, KF, & Prescreener)(LSE, KF, & Prescreener)


DOQ Timing

• Reference Data– USGS Digital Ortho Quarter-Quad (1m GSD)– NIMA Digital Terrain Elevation Data (100m posts)

• Timing Data– SGI Octane– Dual 225MHz R10,000 cpu’s– 512Mb RAM total– Controller, Generator, Worm Combiner thread


NY Intersection Circle Stare


Feb'00 NY Site, Clip A4

0

10

20

30

40

50

60

70

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61

61 frames, EO vs DOQ & DTED1

Gro

und

Erro

r (m

)

beforekftype

before: median: 33.1 90th: 42.4 mean: 32.5 std dev: 8.6 <10m: 0%kf: median: 2.7 90th: 4.4 mean: 3.0 std dev: 1.3 <10m: 100%



Feb'00 NY Site, Clip A4 Timing Stats

0

2

4

6

8

10

12

14

16

18

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61

61 frames, EO vs DOQ & DTED 1

Late

ncy

(sec

)

reg timereg type


mean: 9.5stdev: 2.6


VA 15-Oct Fast Straight Line


Oct'99 VA Site, Clip D2

0

5

10

15

20

25

30

35

40

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63


Gro

und

Erro

r (m

)

beforekftype




Oct'99 VA Site, Clip D2 Timing Stats

0

10

20

30

40

50

60

70

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63


Late

ncy

(sec

)

reg timereg type



NC Suburban Run


Mar'00 NC Site, Clip B4

0

5

10

15

20

25

30

35

40

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55


Gro

und

Erro

r (m

)

beforekftype


NC Suburban Run


Mar'00 NC Site, Clip B4 Timing Stats

0

50

100

150

200

250

300

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55

55 frames, EO vs DOQ & DTED 1

Late

ncy

(sec

)

reg timereg type

NC Suburban Run


DOQ Validation Summary

Rollup - DOQ - Before Registration

0

10

20

30

40

50

60

70

A1 A2 A3 A4 B3 B4 D2 D3 D4

Gro

und

Erro

r (m

)

Rollup - DOQ - After Registration

0

5

10

15

20

25

30

35

A1 A2 A3 A4 B3 B4 D2 D3 D4 G

roun

d Er

ror (

m)

LEGEND:• maximum- 90th percentile- 3rd quartile- median- 1st quartile- 10th percentile• minimum

Note 2X scale changeBefore vs. After


DOQ Validation Summary

11 June 2001 Rollup - DOQ

05

1015202530354045

A1 A2 A3 A4 B3 B4 D2 D3 D4

Gro

und

Erro

r (m

)

0.0%

20.0%

40.0%

60.0%

80.0%

100.0%

120.0%

Succ

ess

(% <

10m

)

Mean Before Mean After Std Dev Before Std Dev After Success Rate

Look Angle GSD # FramesHigh Fine 0High Medium 273High Coarse 6

Moderate Fine 0Moderate Medium 134Moderate Coarse 165

Low Fine 0Low Medium 12Low Coarse 60

DOQ Total: 650

High > 55 Fine .15-.3Mod 35-55 Med .3 - 1Low < 35 Coarse > 1


References

J. K. Bryan, D. M. Bell, A. J. Lee, N. H. Carendar, and F. H. Baker, “A New Image Registration Paradigm”, Electronic Imaging International Conference Proceedings, Boston, MA, Sep. 1993.

D. M. Bell, J. K. Bryan, and S. B. Black, “Mechanism for Registering Digital Images Obtained From Multiple Sensors Having Diverse Image Collection Geometries,” U.S. Pat. No. 5,550,937, Aug. 1996.

J. Hackett, D. Trask, and R. Cannata, “Automated Near-Real Time Registration and GeopositioningBased Upon Rigorous Photogrammetric Modeling,” ASPRS-RTI Conf. Proc., Tampa, FL, 1998.

A. J. Lee, D. M. Bell, and J. M. Needham, “Adjustment of Sensor Geometry Model Parameters Using Digital Imagery Co-registration Process to Reduce Errors in Digital Imagery Geolocation Data,” U.S. Pat. No. 5,995,681, Nov. 1999.

R. Cannata, M. Shah, S. Blask, and J. Van Workum, “Autonomous Video Registration Using Sensor Model Parameter Adjustments,” Proc. 29th Applied Imagery Pattern Recognition Workshop,Washington, D.C., Oct. 2000, pp 215-222.

J. Van Workum and S. Blask, “Adding Precision to Airborne Video with Model Based Registration,” Proc. 2nd Int’l Workshop on Digital and Computational Video, Tampa, FL, Feb. 2001, pp 44-51.

S. G. Blask and J.A. Van Workum, “An Autonomous 3D Photogrammetric Approach to Airborne Video Geo-Registration,” Invited Talk at IEEE Workshop on Video Registration held with The Eighth IEEE International Conference on Computer Vision, Vancouver, BC, Canada, July 13, 2001, http://www.cs.ucf.edu/~vision/workshop/workshop.html.

Documents

An Autonomous 3-D Photogrammetric Approach to …vision.eecs.ucf.edu/workshop/sblask_reg.pdf · An Autonomous 3-D Photogrammetric Approach Steven G. Blask, ... MP7 + MP8 MP9 + + MP10