A Review of Calibration, Validation and 3D Measurements for

Underwater Stereo-video Systems Mark Shortis

School of Mathematical and Geospatial Sciences RMIT University, Melbourne, Australia

ISPRS / CIPA Workshop UNDERWATER 3D RECORDING & MODELING

16-17 APRIL 2015 Piano di Sorrento, Italy

Acknowledgements Euan Harvey and Nader Boutros

University of Western Australia/Curtin University

Alan Williams and Bruce Barker CSIRO Marine and Atmospheric Research

Mike Cappo Australian Institute of Marine Science

Why Bother with Calibration? • Unnecessary for low accuracy applications. • Use a correction lens (Ivanoff &Cherney, 1960). • Use the classic grid approach to take

measurements within a consistent XY system.

nauticalarch.org

Why Bother? 2D+ or Accuracy! • Refraction effects must be modelled to avoid

systematic effects and scale errors.

listverse.com

Why Accurate Size is Important - Biomass

Napier and Goodland, 1997

Why Accurate Size is Important - Age

Napier and Goodland, 1997 Snake River chinook salmon

Beer, 1998

Four sets, Sept 2002, Bowling Green Bay; stereo-rig - 248 measurements

Lower end of 100mm length class

-20 80 180 280 380 480 580 680 780 880 980 2000 2200

Freq

uenc

y

0

10

20

30

40

50Carangids, Nemipterids

Spanish mackerel

School mackerel

Serranids, lutjanids, lethrinids, pomacanthids

Maori wrasse, tuskfish

shark ray

Balistids, lutjanids

NOAA Pacific Islands

A Concise History

1850

1915

1930

1945

First underwater images using glass plates

First underwater movie

O-ring seals make water-tight housings possible

AquaLung invented by Coutseau

1949

1957

1960

Rolleimarin underwater housing

Waterproof 35mm camera invented

First seabed and archaeological mapping

1950s Surges in SCUBA diving, photography, underwater feature movies, exposure of marine environment

1980s

2006

2000s

Stereo-photos used for quantitative analysis and underwater stereo-video adopted by marine science

GoPro revolution starts

First solid state video recording systems

1970 Film stereo cameras with correcting lenses

1990s Digital still cameras arrive and DV tape released

Calibration Techniques

NOAA Hawaii

Turner, 1992

Self or System? Pre- or In-situ? • Relatively few examples of self-calibration using

the object of interest (Schewe et al, 1996). • Pre-calibration with a calibration fixture (Harvey

& Shortis, 1996) or a ‘Manhattan’ object (Turner, 1992) is the typical approach.

• Some mechanism for scale determination must be included (Menna et al, 2013).

• Scale can also be determined from known distances on the calibration fixture.

• Stability of the camera and optical path are critical.

Camera Calibration Algorithms • Absorption by the standard physical

parameter set (Fryer and Fraser, 1986; Harvey and Shortis, 1996; Schewe et al, 1996; Menna et al, 2013)

• Geometric correction (Li et al, 1997)

• Perspective centre shift (Telem and Filin, 2010)

• For stereo-cameras, the relative orientation is

computed using a constrained network solution or from post-processing of network output data.

Pros and Cons Technique Advantages Disadvantages

Correction Lenses Simple solution. Still requires calibration for high accuracy.

Absorption Standard physical parameters, no change to calibration algorithm used in air.

May still be some residual effects that are not modelled accurately.

Geometric Correction In theory can fully correct for refraction for a specific lens cover type.

Complex, two phase solution that is specific to the optical path geometry.

Perspective Shift In theory can fully correct for refraction for a specific lens cover type.

Solution is specific to the optical path geometry. Some physical terms are neglected, assumed to be absorbed by the standard calibration parameters.

Absorption • The principle effect of refraction is radial

distortion about the optical axis. • Small non-radial effects are absorbed by

decentring lens distortion and affinity terms.

Geometric Correction • Li et al 1997: specific

to concave lens covers. • Two phase calibration:

– In air calibration first, standard parameters.

– In water calibration second, 11 lens cover parameters and 2 refractive indices.

– Refractive indices of air and water can be pre-determined or estimated.

Perspective Centre Shift • Telem and Filin 2010: specific solution for a

planar housing/port. • Parameters include:

– standard physical parameters, – refractive indices of glass and water, – distance between the perspective centre and the

port, – tilt and direction of angle between optical axis and

normal to the housing port, and – housing interface thickness.

Perspective Centre Shift (2)

Some parameters are not included, and any remaining signal is absorbed by the standard calibration parameters.

Calibration Fixtures – Image Space • Size is a critical factor, can be partly

compensated by moving around within the FoV. • 3D is a critical factor, compensation as above.

www.vision.caltech.edu

Calibration Fixtures – 3D Space • Size is a critical factor, the fixture must suit the

expected distance range of measured objects. • The fixture should be moved to ‘fill’ the range.

Calibration Results Technique RMS Image

Error (pixels) RMS XYZ Error

(mm) Proportional Error

Shortis & Harvey, 1996…. absorption

0.1 - 0.3 0.1 - 0.5 1:3000 – 1:15000

Schewe et al 1996 absorption

0.3 40-200 1 : 500

Li et al 1997 geometric correction

(1.0) 10 1 : 210

Sedlazeck & Koch, 2009…. geometric correction

0.2 5 -

Telem & Filin 2010 perspective shift

0.3 2.0 1 : 1000

Menna et al 2013 absorption

0.2 – 0.25 1.9 1 : 32000

Accuracy Validation • RMS image residual not a good indicator. • Corrections to target XYZs or lengths are useful. • Better to use scale bar checks or silhouette

measurements throughout FoV.

Validation Results Technique Validation Percentage Error

Shortis & Harvey, 1996… absorption

Length measurement of silhouettes or rods throughout

the volume

0.2 – 0.7 %

Petrell et al, 1997 lens distortion grid

Calliper measurements of Chinook Salmon

1.5 %

Harvey et al, 2003 absorption

Calliper measurements of Southern Bluefin Tuna

0.2 %

Telem & Filin 2010 perspective shift

Flat reference plate and straight line re-construction

0.4 %

Menna et al 2013 absorption

Similarity transformation between above water and

below water networks

0.3 %

Wehkamp & Fischer 2014 Matlab and radial lens distortion

Distances on checkerboard 0.9 – 1.5 %

Calibration Stability • Estimated using

parameter changes between sequential calibrations.

• Lengths on the fixture ensures scale stability.

• Calibration applies to the entire optical path.

• Camera-port interface is a critical sensitivity.

0

5

10

15

20

0 2 4 6 8 10 12 14 16

Cha

nge

Sign

ifica

nce

Calibration Date (Days)

Open Water Right Camera Calibrations

PPx

PPy

PD

Rad Distn

Dec Distn

Affinity

Case Study • 3D cube using a self-calibrating

photogrammetric network compared with A3 and A4 checkerboards using the Bouget Matlab toolbox solution (Boutros et al, 2015).

• Stereo-video system of two Sony HDR-CX700 HD format camcorders with purpose-built housings designed to minimise movement.

• RO included as an integral part of the tests. • Also compared a range of three base

separations: 150mm, 400mm and 830mm.

Diver and AUV Systems Study Subject Range (m) Base (mm)

Seiler et al 2012 Marine fish < 2 78

Williams et al 2010 Benthic habitat < 2 78

Shortis et al 2000 Shellfish 0.5 250

Abdo et al 2006 Sponges 0.5 336

Ruff et al Captive fish 1 – 2 500

Langlois et al 2012 Reef fish 0.5 – 8 700

Santana-Garcon et al 2014 Pelagic fish 1 – 10 700

Harvey et al 2003 Southern Bluefin Tuna 2 – 8 800

Seager 2014 Whale sharks 10 – 12 900

Harvey & Shortis 1995 Reef fish and habitat 5 1400

Length Measurement Accuracy

Calibration Parameters

Impact of Base Separation

Effect of Calibration on Biomass Estimates

Percentage error for a hypothetical 400mm snapper (Pagrus auratus)

Range (m) Cube A3 A4

2 0.00 0.91 2.12

3 0.49 0.09 0.54

4 1.07 0.92 0.02

5 1.49 2.67 10.20

6 1.88 6.03 3.57

7 2.45 7.98 16.01

8 2.03 12.18 42.90

Effect of Base on Biomass Estimates Range (m) 800mm 400mm 150mm

2 0.22 0.82 1.35

3 1.42 5.06 1.60

4 0.54 0.76 3.32

5 1.27 0.78 4.29

6 1.54 1.22 6.77

7 1.87 0.47 3.31

8 1.38 0.83 2.83

9 1.44 0.70 6.66

10 1.45 4.31 7.67

11 1.69 2.60 35.06

12 0.32 5.41 19.86

Percentage error for a hypothetical 400mm snapper (Pagrus auratus)

Analysis

• A 3D calibration object clearly produces improved accuracy, precision and reliability.

• Short baselines give surprisingly good results, but deteriorate quickly at longer ranges.

Future Developments • GoPro revolution is generated a need for rapid

calibration of different modes of operation. • Digital SLR cameras require (separate)

calibration in still and movie modes. • Different algorithms are likely to suit different

housings. • Simple, glass or acrylic plane surface

waterproof housings may suit absorption. • Curved acrylic housings may produce

improved results with geometric corrections.

Questions? Principal References: Boutros, N., Harvey, E. S., Oldham, C. and Shortis, M. R., 2015. Calibration and

configuration of underwater stereo-video systems for applications in marine ecology. In press for Limnology and Oceanography: Methods.

Harvey, E. S. and Shortis, M. R., 1996. A system for stereo-video measurement of sub-tidal organisms. Marine Technology Society Journal, 29(4): 10-22.

Ivanoff, A., and Cherney, P., 1960. Correcting lenses for underwater use. Journal of the Society of Motion Picture and Television Engineers, 69(4):264-266.

Li, R., Li, H., Zou, W, Smith, R. G. and Curran, T. A., 1997. Quantitative photogrammetric analysis of digital underwater video imagery. IEEE Journal of Oceanic Engineering, 22(2): 364-375.

Menna, F., Nocerino, E., Troisi, S. and Remondino, F., 2013. A photogrammetric approach to survey floating and semi-submerged objects. Videometrics, Range Imaging, and Applications XII, SPIE Vol. 8791: paper 87910H.

Schewe, H., Moncreiff, E. and Gruendig, L., 1996. Improvement of fish farm pen design using computational structural modelling and large-scale underwater photogrammetry. International Archives Photogrammetry and Remote Sensing, 31(B5) : 524-529.

Telem, G. and Filin, S., 2010. Photogrammetric modeling of underwater environments. ISPRS Journal of Photogrammetry and Remote Sensing, 65(5): 433-444.

Turner, J., 1992. The development of an operational digital photogrammetric system for the North Sea oil and gas industry. Videometrics II, SPIE Vol. 2067, pp 136-144.