Towards Qualitative Vision: Motion Parallax

Andrew Blake, Roberto Cipolla &: Andrew Zisserman

Department of Engineering Science *University of Oxford

Oxford, 0X1 3PJ, U K

RELATIVE DEPTH

We propose the use of motion parallax - therelative image motion of nearby image points- as a robust geometric cue for the compu-tation of relative depth and surface curvatureon specular surfaces and at extremal boundaries.

PARALLAX IS A ROBUST CUE

A robot vehicle moving under visual guidanceneeds to compute approximate geometry of ob-stacles in its environment. It is unreasonable toassume that egomotion is known to the sort ofprecision that is available for a camera mountedon a high quality robot arm. Generally a nomi-nal estimate for egomotion is available. One pos-sibility is to refine this estimate using optic flowdata (Harris, 1987). Alternatively, the problemcan be turned on its head: what geometric infor-mation remains stable under perturbation of as-sumed egomotion? This question has been ad-dressed by Koenderink and van Doom (1977), Nel-son and Aloimonos (1988) and Verri et al. (1989),in the case of continuous motion fields and, in thedomain of stereoscopic vision, by Weinshall (1990).Part of the answer, we claim, lies in the use ofmotion parallax as a geometric cue. Motion par-allax, which is a relative measure of the positionsof two points, can be very much more robust asa cue than the absolute position of a single point.This is true for computation of relative depth, cur-vature on specular surfaces and curvature on ex-tremal boundaries.

"The authors acknowledge the support of SERC, IBMUK Scientific Centre and Esprit BRA 3274 (FIRST).

Longuet-Higgins and Prazdny (1980) showed thatwhereas the structure of an image motion field isgenerally confounded by viewer rotation, motionparallax does not suffer that problem. Using theconvention of Maybank (1985) that the image isprojected onto a unit sphere through its centre, apoint r on a visible surface projects to a vectorQ(t) on the image sphere:

r = X(t)R(t)Q(t), (1)

where, at time 2, v(t) is the position of the viewer,\(t) is the distance along the ray from the viewerto the point r and R(t) is a rotation operator de-scribing the orientation of the camera frame rela-tive to the world frame. The motion is assumedrigid, involving translational motion U and rota-tional motion fi defined by:

U = v and (fix) = R

where' denotes differentiation with respect to timeand X denotes vector product. The image motionof the projection of Q of the point r is then

i (2)

The rotational contribution is apparent, and hardto extricate, although numerous solutions to thatproblem have, of course, been devised.

One simple solution involves parallax. A secondpoint Q* is introduced which, instantaneously, co-incides with the first, so that Q(0) = Q*(0). Thenthe relative displacement of the two image pointsis A — Q - Q* and the parallax is its temporalderivative A. It is straightforward to show that

(3)= ( j - - ^ ) (U x Q) x Q,

which is independent of rotation. This means thatrelative inverse depth I /A- I/A* can be computed

115

from parallax more robustly than absolute depthcan be recovered from image motion. It is, the-oretically, entirely immune to errors in estimatedviewer rotation. In practice, of course, there isa residual sensitivity owing to the fact that Q,Q*will not coincide exactly at the instant of measure-ment.

Rieger and Lawton (1985) have implemented ascheme using motion parallax for robust estima-tion of relative depth and direction of translationfrom real image sequences. They also shown howthe results degrade with increasing separation ofthe 2 points, Q,Q*.

CURVATURE OF SPECULAR SUR-FACES

It has been shown (Blake and Brelstaff, 1988) thatparallax of a specularity - relative motion of ahighlight and nearby surface feature - is a robustcue for surface curvature. It is more robust thanstereo-based curvature estimates because spatialderivatives need not be computed in the specularcase. It appears that human vision is capable of us-ing this cue (Blake and Biilthoff, 1990). An earlierqualitative model dealt only with creation and an-nihilation of specularities at parabolic lines (Koen-derink and van Doom, 1980). This is structurallya very robust cue but somewhat sparse. Specu-lar parallax cues are more common, being stablewith respect to viewpoint. Surface curvature in theform of the Weingarten map W (Thorpe 1979) isconstrained by observed parallax A:

2\2WVlA = -V2U. (4)

Here V\ and V2 are dimensionless projection op-erators involving combinations of projections intoimage and surface tangent planes. A derivation ispresented in (Blake et al,1988,1990). The detailsare not important, but suffice it to say that V\and V2 reduce to identity operators for the spe-cial case of fronto-parallel viewing. Note that thisequation is correct in the limit that the source andviewer are distant from the surface. Thus the sin-gular effects of the crossing of the caustic by theviewer are neglected. However the full model iseasily obtainable (Blake and Brelstaff, 1988).

Clearly the constraint is independent of, and hence

robust to rotational motion fi. If instead of par-allax, the absolute image motion of the specu-larity were substituted the result would be sen-sitive to rotation and, even worse, sensitivity totranslational motion U would be greatly enhanced.However, even with parallax there is a remaining,though linear and hence well-conditioned, depen-dence on viewer motion U. There is also an im-plied (by V\,V'i) dependence on estimated sourceposition. This too can be eliminated by using astill more qualitative measure than parallax A,namely the sign of its epipolar component (Bolleset al. 1987). This sign is sufficient for con-vex/concave discrimination, is independent of lightsource position and the magnitude of of transla-tional velocity U, depending only on the directionof U to define epipolar lines on the image sphere(Zisserman et al. 1989). Human observers are ableto use this cue for disambiguation of reversible fig-ures (Blake and Biilthoff 1990).

SURFACE CURVATURE ON EX-TREMAL BOUNDARIES

Building on earlier work (Marr 1977, Barrow andTenenbaum 1978, Koenderink 1984, Giblin andWeiss 1987) we have developed a general model ofthe inference of surface curvature from the defor-mation of apparent contours (Blake and Cipolla1989). As a viewer moves past a curved sur-face, apparent contours, the projections of ex-tremal boundaries, move non-rigidly on the imagesphere. Their motion and deformation can be usedto characterise local surface shape including cur-vature, along the extremal boundaries. It can alsobe used, of course, to discriminate between fixedsurface features and extremal boundares.

The characterisation turns out to be robust onlyif a parallax-based measure is used. Intuitivelyit is relatively difficult to judge, moving arounda smooth, featureless object, how large the nor-mal curvature across the extremal boundary is.However, this judgment is much easier to makefor objects which have at least a few surface fea-tures. Under small viewer-motions, features are"sucked" over the extremal boundary, at a ratewhich depends on surface curvature. Our theoret-ical findings exactly reflect the intuition that the"sucking" effect is a reliable indicator of normal

116

curvature, regardless of the exact details of theviewer's motion.

As before, parallax is denned as A where A =Q — Q*, and now Q is a point on the apparentcontour whereas Q* is the image of a reference fea-ture on the surface. It is assumed that the surfacereference point lies close (in 3D space) to the ex-tremal boundary. The normal curvature K of thesurface along the line of sight can be computedfrom parallax as follows:

where n is the normal (in the tangent plane to theimage sphere) to the apparent contour. Normalcurvature K, together with curvature of the appar-ent contour, is in fact sufficient to compute full sur-face curvature at a point (Blake and Cipolla 1989).Again the estimate of K is robust in the sense thatit is independent of rotational velocity £1. Bet-ter than that, curvature estimates based on abso-lute image motion and acceleration is also sensitiveto linear and rotational acceleration, somethingwhich is eliminated when the "rate of parallax" Ais used as above. The sign of n determines the "sid-edness" of the extremal boundary - on which sideof the image contour lies the curved surface. Thisqualitative sidedness cue depends only on knowl-edge of the direction of translational motion, to es-tablish the epipolar lines, as before. Finally, ratiosof normal curvatures are completely independentof any assumption about viewer motion! Termsdepending on absolute depth and translational ve-locity are cancelled out in equation (5). As before,independence degrades as the assumption that thereference point is close to the extremal boundaryis relaxed. The degradation can be theoreticallypredicted and is analysed in (Cipolla and Blake90).

Results in figures 3,4,5 illustrate that the theoret-ical robustness of rate-of-parallax are borne outin practice. They show the sensitivity of the esti-mated radius of curvature, (R — 1/K) at a pointon an extremal boundary (figure 2) computed fromknown viewer motion when an error is introducedin the camera positions and orientations. Esti-mates of curvature not using parallax require pre-cise knowledge of viewer motion (1 part in 1000).This sensitivity is reduced by an order of magni-tude if parallax measurements are used instead. It

is further reduced by another order of magnitudefor ratios of parallax measurements. The residualsensitivity to rotation and velocity is, as predicted,due the finite separation of the point on the ex-tremal boundary and the surface marking.

CONCLUSIONS

We believe that the use of motion parallax in therecovery of surface geometry from surface texture,specularities and apparent contours represents asignificant step in the development of practicaltechniques for robust, qualitative 3D vision.

We are currently working on the realtime imple-mentation of algorithms using motion parallax foruse in the active exploration of the 3D geometryof visible surfaces for navigation. Preliminary re-sults involving the tracking of image contours andspecularities; discrimination between fixed and ex-tremal features; and the recovery of strips of sur-faces in the vicinity of an extremal boundary orspecularity are described in (Cipolla and Blake,1990) and (Zisserman et al, 1990).

117

Figure 1. Sample of monocular image sequence ofmotion of specularities across the curved surface of aJapanese cup with viewer motion. 4 images of thesequence (magnified window) showing the relativemotion (parallax) of a point specularity (Q) and fixedsurface marking (Q*) are shown in Figure lb.Parallax measurements can be used to determineSurface curvature and normal along the path followedby the specularity as the viewer moves. A morequalitative measure is the sign of the epipolarcomponent of the parallax measurement. With viewermotion the specularity moves in opposite directionsfor concave and convex surfaces.

Figure 2. Sample of monocular image sequenceshowing the image motion of apparent contours withviewer motion. 4 images of the sequence (magnifiedwindow) showing the relative motion between theapparent contour (projection of the extremalboundary) and a nearby surface marking (shown as across) are shown in Figure 2b. The relative imagemotion as the feature moves away from the extremalboundary can be used for the robust estimation ofsurface curvature.

Figure lb. Relative motion of specularity and nearbysurface marking

Figure 2b. Relative motion of an apparent contour

118

a

200-

150-

100 -

so •

yS S ^ 50

y S > S 100 •

R/inm / B

/ / / A

position error/mm

bB

A . ^ - \ ^ 1M

5 ••* 3 2 -1

rotation error/mrad

-100 •

-150 •

R/mm

Figure 3. Sensitivity of curvature estimated from absolute measurements to. errors in.rnotign.The radius of curvature (mm) for both a point on a surface marking (A) and a point on an extremalboundary (B) is plotted against error in the estimate of position (a) and orientation (b) of the camerafor view 2. The estimation is very sensitive and a perturbation of lmm in position produces an error of190% in the estimated radius of curvature for the point on the extremal boundary. A perturbation oflmrad in rotation about an axis defined by the epipolar plane produces an error of 70%.

a?50

200 -

ISO -

100 -

— — — — — 5 t t j

-2.5 Zd 1.5 -1.0 OS

-50 -

100 -

• 150 •

R /mm

0.5 1.0 1-5 7.0 2.S

error/mm

b

5 -4

250-

200 •

150 -

100 •

3 -2 1

50 •

-100 •

-150 •

R/mm

—-——

1

• — —

2 3 * 5

eiror/mrad

Figure 4. Sensitivity of differential curvatureThe difference in radii of curvature between a point on the extremal boundary and the nearby surfacemarking is plotted against error in the position (a) and orientation (b) of the camera for view 2. Thesensitivity is reduced by an order of magnitude to 17% per mm error and 8% per mrad errorrespectively.

aMO -

200

150 •

100

-2.5 -2.0 .1.5 -1.0 -OS

-50 •

-100 •

•150 -

R /mm

OS 1.0 1.5 70 Z5

error/mm

b250-

200 -

150 •

100 •

•5 -4 -3 -2 -1

-SO •

100 -

-150 -

R/mm

1 3 3 4 5

error/mrad

Figure 5. Sensitivity of ratio of differential curvaturesThe ratio of differential curvatures measurements made between 2 points on an extremal boundary andthe same nearby surface marking is plotted against error in the position (a) and orientation (b) of thecamera for view 2. The sensitivity is further reduced by an order of magnitude to 1.5% error for a lmmerror in position and 1.1% error for lmrad error in rotation. The vertical axes are scaled by the actualcurvature for comparision with figures 3 and 4.

119

References

[1] H.G. Barrow and J.M. Tenenbaum. Re-covering Intrinsic Scene Characteristics fromImages. A.I Center Technical Report 157, SRIInternational, (1978).

[2] A. Blake and R. Cipolla. Robust Esti-mation of Surface Curvature from Deforma-tion of Apparent Contours. Technical Re-port OUEL 1787/89, University of Oxford,1989. Also Proc. 1st European Conf. Com-puter Vision, Springer-Verlag, (1990).

[3] A. Blake and H. Biilthoff. "Does the brainknow the physics of specular reflection?" Na-ture 343, 165-168 (1990).

[4] A. Blake and H. Biilthoff. "Shapefrom Specularities: Computation and Psy-chophysics" (to be published).

[5] A. Blake and G.J. Brelstaff. "Geometryfrom Specularities." In Proc. 2nd Int. Conf.Computer Vision, 394-403 (1988).

[6] R.C. Bolles, H.H. Baker, and D.H. Ma-rimont. " Epipolar-plane image analysis: anapproach to determining structure." Interna-tional Journal of Computer Vision, vol.1:7-55, (1987).

[7] R. Cipolla and A. Blake. "The dynamicanalysis of apparent contours." Technical Re-port OUEL 1828/90, University of Oxford,(To appear in Proc. 3rd Int. Conf. ComputerVision, Osaka)(l990).

[8] P. Giblin and R. Weiss. " Reconstructionof surfaces from profiles." In Proc. 1st Int.Conf. on Computer Vision, 136-144, (1987).

[9] C.G. Harris. "Determination of ego - motionfrom matched points." In 3rd Alvey VisionConference, 189-192, (1987).

[10] J.J. Koenderink and A.J. van Doom,"How an ambulant observer can construct amodel of the environment from the geomet-ric structure of the visual inflow." Kibernetic1977 G., eds. G. Hauske and E. Butendant,Oldenbourg, Miinchen (1977).

[11] J.J. Koenderink and A.J. van Doom,"Photometric invariants related to solidshape." Optica Ada, 27, 7, 981-996 (1980).

[12] J.J. Koenderink. "What does the occlud-ing contour tell us about solid shape?" Per-ception, 13:321-330,(1984).

[13] H.C. Longuet-Higgins and K. Pradzny."The interpretation of a moving retinal im-age." Proc. Royal Soc. London, B208:385-397, (1980).

[14] D. Marr. " Analysis of occluding con-tour." Proc. Royal Soc. London, 197:441-475,(1977).

[15] S.J. Maybank. "The angular velocity asso-ciated with the optical flow field arising frommotion through a rigid environment." Proc.Royal Soc. London, A401-.317-326, (1985).

[16] R.C. Nelson and J. Aloimonos, "Usingflow field divergence for obstacle avoidance:towards qualitative vision." Proc. 2nd Int.Conf. Computer Vision, (1988).

[17] J.H. Rieger and D.L. Lawton. "Process-ing differential image motion." J. Optical Soc.of America, vol A2, no. 2, 354-359 (1985).

[18] J.A. Thorpe. Elementary topics in differ-ential geometry. Springer-Verlag, New York(1979).

[19] A. Verri, F. Girosi and V. Torre. "Mathe-matical properties of the two-dimensional mo-tion field: from singular points to motionparameters." JOSA vol A6, no. 5, 698-712,(1989).

[20] D. Weinshall. " Qualitative depth fromstereo with applications." Computer Vision,Graphics, and Image Processing, 49, 222-241,(1990).

[21] A. Zisserman, P. Giblin and A. Blake."The information available to a moving ob-server from specularities." Image and VisionComputing, 7, 38-42, (1989).

[22] A. Zisserman and A. Blake "Shape fromtracked specular motion." Technical Report,University of Oxford (in preparation)(1990)

120