Recovering Partial 3D Wire Frames Descriptions from Stereo DataStephen B Pollard, John Porrill and John E W Mayhew

AIVRUUniversity of Sheffield

We describe the design of modules in the current ver-sion of the TINA stereo based 3D vision systemresponsible for the recovery of geometric descriptionsand their subsequent integration into partial wireframe models. The approach differs considerably fromthat described in an earlier version of the TINA sys-tem [2, 3]. Our strategy for the construction of partialwire frame descriptions is broken into the followingthree processing stages:

(1) Stereo processing. This is edge based and usescamera calibration information to provide arestriction on allowable matches between edgesextracted from the left and right images. Thealgorithm does not, however, at this stagerecover an accurate estimate of disparity ofeach pair of matched edges.

(2) Geometrical description. This relies on theidentification of higher level 2D primitives fromthe edge strings in one, other or both imagesand their subsequent combination with stereomatching, and calibration data to allow therecovery of 3D primitives.

(3) Wire frame generation. Based upon 2D and 3Dproximity and connectivity heuristics potentialwire frame components are identified.

1. Introduction

Feature based stereo (either binocular or trinocular)provides one of the simplest, best understood andmost robust methods for the passive recovery of threedimensional information from image data. Further-more, given the availability of suitable a priori infor-mation concerning the structure of the imaging sys-tem, geometrical descriptions recovered from stereoare metrical if (depending upon the care taken in theirrecovery) not always accurate. However, for manyrobotic tasks the availability of unstructured depthdata alone is not sufficient for their effective comple-tion. In general the data must be structured in someway to be of use.

This paper is concerned with the interpretation ofedge based stereo data with a view to the construction

of partial 3D wire-frames. The latter is a descriptionthat makes explicit potential wire frame primitivesand hypothesised connections between them. It doesnot however involve the completion of wire frames tothe status of boundary descriptions of physicallyrealisable objects. As such partial wire frames areallowed to include descriptions that arise fromchanges in surface reflectance properties whilst fullwire frames are not. We conjecture that full wireframe descriptions can only be reliably recovered inthe presence of domain knowledge. Partial wireframes on the other hand are built purely bottom upusing only local heuristics.

Of course partial wire frames descriptors will ofthemselves be of considerable use. For example weshall show that Geomstat [1] a software package thatuses statistical estimation theory to enforce geometri-cal constraints (such as intersection, parallelism,orthogonality, continuation, coplanarity, etc) can beeffectively driven over partial wire-frame data.

2. Edge Based Stereo

Edges are obtained, to sub pixel acuity, from grey-level images by a single scale high frequency applica-tion of the Canny edge operator [4]. The effect of theoperator is essentially to identify contours correspond-ing to above threshold maxima of image intensity gra-dients. Noise suppression and scale selection areachieved by a pre-smoothing stage involving convolu-tion with a gaussian mask (the size of the mask deter-mining the degree of smoothing). The high frequencyoperator used here employs a gaussian mask of sigma1.0 (diameter of 9 pixels when thresholded at 0.001of the peak value). Fortunately the two dimensionalgaussian smoothing can be achieved through two 1dimensional convolutions (eg. first along the rows andthen the columns). Further improvements inefficiency can be achieved through the use of recur-sive filters [4, 5].

Recent and more effective versions of our stereo algo-rithm, called PMF, are used to match correspondingedges [6]. Essentially matches between edges fromthe left and right images are preferred if they mutu-ally support each other through a disparity gradientconstraint (ie. that the ratio of the difference in their

31 BMVC 1990 doi:10.5244/C.4.8

disparity to their physical separation is below a thres-hold of 0.5) and if they satisfy a number of higherlevel grouping constraints, most notably uniqueness,ordering (along epipolars) and figural continuity. Ini-tially individual matches are restricted to be of similarcontrast (within a factor of 3) and similar orientations(consistent with a disparity gradient limit; as a conse-quence of which edges perpendicular to the epipolardirection are permitted to reorientate more extremelythan those runing along it).

Using information about the geometry of the camera'sobtained from a calibration stage allows the matchingproblem to be reduced to a one dimensional searchalong corresponding epipolars. Furthermore if theedge maps are transformed to an alternative parallelcamera geometry that is equivalent to the original, theepipolar constraint becomes a same raster constraintwith obvious benefit in reducing the complexity of thestereo matching algorithm.

The results of this stage of processing are connectedstrings of edge primitives from the left image some ofwhich have associated with them edge primitives withwhich they are thought to match in the right image.It is possible for connected edges to run along theepipolar (raster) hence (especially if the edge direc-tion is almost horizontal) it is sometimes difficult (andno attempt is made here) to resolve matches uniquelyat this stage.

3. Obtaining Geometrical descriptions

While the availability of suitable estimates of the epi-polar geometry and/or the intrinsic and extrinsic cam-era parameters is fundamental to the accuraterecovery of qualitative and quantitative shape descrip-tors from stereo; practical, robust and generic visualcapabilities require much attention to the method usedto incorporate disparity information in the construc-tion of the three dimensional descriptors themselves.

We begin this section by considering three strategiesby which stereo edge data can be combined (assum-ing known epipolar geometry) to recover 3D sceneprimitives. In order to clarify the issues involved weplace each method in the context of recovering aplane curve from a string of matched edge data (seealso figure 1).

The scheme used in the early versions of TINA [7] isto begin by obtaining estimates in a disparity spacespace (chosen to be the unbiased space [XI, Xr, Y])from the sub-pixel estimations of individual edgecorrespondences (taking account of their off-epipolardifferences). Subsequently (and after suitable segmen-tation) a plane is fitted to the disparity data by orthog-onal regression. Unfortunately this process is unable

to recover from the initial sub-optimal estimate of theindividual disparity values.

Figure 1.

The figure illustrates three possible strategies for thecombination of stereo matching data to recover planecurve descriptors. In each sub pixel edge locationsare shown in the left and right image as dots. In Aleft and right image primitives are combined andindividual disparity estimates obtained prior to fitting.In B optimal fitting is first performed in the left imagewith a subsequent recovery of the affine transformrequired to minimise the sum squared error of righthand edge points to the transformed curve. In C (thepreferred method) consider the epipolar that passesthough a point in the right image and the correspond-ing epipolar in the left (labeled el and er respec-tively). In general no left edgel will not lie exactly onel and hence the disparity measure will be sub-optimal. The edge comprised of the edgels in the leftimage can be approximated by the curve C (if possi-ble and computationally plausible, optimally). Theoptimal estimate of disparity is obtained from theintersection of the epipolar el with C.

32

In the method proposed by Cooper, Hung and Taubin[8], edges are initially brought into correspondence atthe level of extended edge strings. Given surface andcurve parameterisation schemes (in their case planeand polynomials respectively) maximum-likelihoodestimation is employed to [implicitly] refine matchselection to give optimal estimates of both the surfaceand curve parameters. In practice curve estimation inone or other image precedes the stage of surface esti-mation and the parameterisation of the latter is interms of an affine transforms between the images(whilst in theory it could, the affine transform is notrestricted to conform to the known epipolargeometry). The primary disadvantage of this approach(beyond purely computational considerations) is itslack of uniformity with respect to possible curve andsurface parameterisations.

Here we exploit a more pragmatic, though notoptimal, solution to the problem which lies betweenthose described above. As with Cooper et al's methodhigh level feature grouping (edge strings, straightlines, conic sections, cubic splines, segmentedregions, etc) is performed prior to disparity detection.Optimal estimates of disparity are obtained along the2D geometrical descriptors for each matched edgepoint from the intersection of the descriptor with theepipolar corresponding to the sub-pixel location of thematched point. The optimal disparity estimates arethen combined in a second stage of fitting (in thiscase to a plane) in 2D arc position against disparity.Hence we assume only the availability of a genericedge matching capability that is able to bring intorough correspondence edge primitives from the leftand right images.

In addition to its uniformity over curve and surfacedescriptions (which are illustrated in the next section)there are a number of additional advantages to thisapproach over either or both of the above:

(1) The 2D descriptions are reasonably robust.

(2) The 2D descriptions can be obtained economi-cally.

(3) Edge matching is reasonably generic.

(4) Disparity to segment fitting is optimal.

(5) Geometrical constraints (eg. planarity) can beexploited easily.

(6) The 2D descriptions remain available forunmatched features.

3.1. Describing Edge Strings

In this section we describe methods for the segmenta-tion, recovery and description of various 2 and 3Dprimitives.

The algorithm to find straight lines, for example, usesa recursive fit and segment strategy (see figure 2).Segmentation points are included when the underlyingedge string deviates from the current line fit. Theactual fit (first in the image and then in arc lengthagainst disparity) is computed by orthogonal regres-sion. Notice that it is simple matter to eliminateoccasional erroneous matches if their disparity valuesare inconsistent with the majority. A good example ofperformance upon objects for which straight lineapproximation is appropriate is given in figure 6 parts(a) and (b).

Figure 2.

Polygonal approximation by recursive fit and segmen-tation. In A the initial sampling of an edge string isshown. The longest linear section is found byefficient heuristic search and fitted by orthogonalregressions and is shown in B. Further recursion atthis sampling scale recovers no further significantstraight sections and the remaining string this isfurther sub sampled (also shown in B). Continuedrecursion at the new scale recovers further straightsections shown in C and D. This method has anumber of advantages over traditional polygonalapproximation and is easily extended to other primi-tive types.

33

Elliptical sections and their confidence envelopes areobtained optimally using a bias corrected Kalmanfilter [9]. However a reasonably large quadrant of theellipse is required for accurate description. A suboptimal (but acceptable) estimate for the ellipse in 3Dis obtained through a weighted least square plane fitthrough the disparity data. Weights are appliedaccording to the orientation of the ellipse at the pointat which the disparity measure is obtained. In generalorientations closer to the epipolar direction are subjectto greater pixelation and epipolar calibration errorthan those orthogonal to it.

Generic plane curves are be treated in a fashionanalogous to that for ellipses. The principal issuesare those of representation, segmentation and combi-nation of the 2D and 3D curve descriptors. Thedesired representation should balance the competingproperties of economy, power and uniformity(between the descriptions in the 2D image, disparityspace and the 3D scene). The results presented belowwere obtained using standard cubic splines.

; • • . > ' /

•7 •' /)

p

/ / """N"—Si1,'.

" ^ ^ ' • • '

Figure 3.

A is a stereo image pair [512 pixels square] of a tri-ple of machined parts. In B are the canny edge pointsidentified in each image, and in C are respectively onthe left and right hand sides a polygonal approxima-tion and a few extended edge strings obtained fromthe left image.

Figure 4.

A shows depth data to approximate sub pixel locationfor uniquely matched (1 to 1) edge points. In Bstraight line, and selected conic and planer curvedescriptions of the scene are viewed from above. Cand D show just the 3D straight line approximationsfrom two other viewpoints. Whilst straight line sec-tions provide some form of description they are frag-mented and noisy for these kinds of object. E and Fshow curve sections from the same viewpoints with afew straight sections included to provide context.

34

Examples of straight line, conic and plane curvesobtained from machined objects are given in figures3, 4 and 5.

We do not advocate that simple unique descriptorscan be recovered for all edge strings on the basis ofbottom up processing alone. It may be sensible torecover multiple descriptions of the same data foredge sections that are close to threshold boundaries interms of their chosen description, for example a spacecurve may also be described as a sequence of planecurves or even its polygonal approximation. Theappropriate descriptor may only become apparent athigher levels in the processing hierarchy.

Figure 5.

3 widgets. Edge data and connected strings in A andB respectively. Two view points of the recovered 3Ddata (straights and ellipses) in C and D.

4. Building Partial Wire Frames

We have also found the generation of partial wireframe descriptions to be easier, computationallycheaper and more robust when based upon the imageprojection of an object rather than the three dimen-sional description of the identified image primitivesalone. Consider the object in figure 6. We currentlylimit wire frame descriptors to linear sections alone(though the vocabulary of the system that constructswire frames is designed to be easily extended toinclude other types of primitive). It is straight forward

to choose simple neighbourhood criteria in the 2Dimage as it is less vulnerable to noise and featuredrop out; these form the basis of potential connectivi-ties. Each line section is treated as a triple whichincludes the line itself and its two end points, thismakes it straight forward to differentiate connectionsbetween the ends of lines (potential vertices, etc) andother forms of connection (parallelism, tee junctions,etc). A few simple heuristics in the image data can beused to identify various vertex and connectivity (eg.collinearity) types. Vertices in the figure wererecovered using the following simple strategy:

(1) Identify all potential vertex connectors (PVC)between the ends of line sections. That is:within a local neighbourhood; close to point ofline intersection; closer than the other end ofthe same line to the point of intersection.

(2) Consider isolated connectivity graphs amongstPVC's.

(3) Find the computationally trivial simple pairwisevertices first That is connectivity graphs thatconsist of a single PVC.

(4) Now find high order vertices (trihedral verticesand above). These can only arise from mutuallyconsistent cliques of cardinality at least 3identified in the graph structure. Additional con-sistency in terms of the actual location of thevertex is also enforced (that is the location ofthe intersection points identified between allconstituent line pairs must be in general agree-ment). PVC's included in such vertices areremoved from their graphs (this may causesome extra fragmentation, hence stage 3 isrepeated on them).

(5) Finally, identify the best of the remaining pair-wise vertices. Where best is defined as theminimum sum of distances from the end pointsto the point of intersection. Each end point isallowed to occur in only a single such vertex.

Vertices thus identified (see the figure) can be inter-preted amongst the related 3D primitives (if theyexist). Furthermore extra 3D primitives can behypothesised where their existence is implied by the2D partial wire frame (for example a 3D line can begenerated between a pair of 3D vertices even thoughit may only exist in the 2D description itself).

5. Conclusions and Future Directions

We have described those components of the currentversions of TINA system responsible for the recoveryof partial wire frame descriptions. Whilst the work inis far from complete, it does identity a direction in

35

which to go in order to recover good quality wireframe geometrical descriptions from stereo data.Much work is required in the 2D image to identifymeaningful events in the scene in order to applyappropriate constraints to the interpretation of thethree dimensional descriptions.

As discussed above we are in the process of combingthe bottom up knowledge free strategies presentedhere with a top down domain-dependent systemresponsible for inferring completed wire framedescriptions. Domain knowledge for generic objectclasses (houses are the classic example) is representedas frames. The completion process is then cast as amatching problem and slot filling task of the databaseagainst the partial wire frame description. This workis as yet too preliminary to discuss in any greaterdetail.

•

A ^

• ^ • ,/ :

\

/

y

/ I /

-

• • —

B

i

> < /

A polygonal scene suitable for primitive partial wireframe description. A and B are the recovered 2 and 3dimensional straight line approximations respectively.Notice the less complete and more fragmented natureof the 3D data. Connectors implying potential vertexlocations recovered on the basis of 2D neighbour-hoods are shown in C and D. Cleaned vertices (byGeomstat) found by the method given in the text areshown in E and F. The interpretation of the 2D ver-tices amongst the 3D data allows extra 3D primitivesto be hypothesised.

References

1. Porrill J, Pollard S B, and Mayhew J E W (1987)The optimal combination of multiple sensorsincluding stereo vision, and Image and VisionComputing, Vol 2, No 5 174-180.

2. Porrill J, Pollard S B, Pridmore T P, Bowen J,Mayhew J E W, and Frisby J P (1987) TINA:The Sheffield AIVRU vision system, IJCAI 9,Milan 1138-1144.

3. Pollard S B, Pridmore T P, Porrill J, Mayhew J EW, and Frisby J P (1989) Geometrical model-ling from multiple stereo views, InternationalJournal of Robotics Research, vol 8, No 4, 3-32.

4. Canny J. F. (1983) Finding edges and lines inimages, MIT AI Lab. Memo 720.

5. Deriche R (1990) Fast algorithms for low-levelvision, PAMI 12 No 1, 78-91.

6. Pollard S B, Mayhew J E W and Frisby J P (1985)PMF: A stereo correspondence algorithm usinga disparity gradient limit, Perception 14, 449-470.

7. Pridmore T.P., J. Porrill and J.E.W. Mayhew(1987), Segmentation and description of binocu-larly viewed contours, Image and Vision Com-puting, Vol 5, No 2, 132-138.

8. Cooper Hung and Taubin (1989) PAMI.

9. Porrill J. (1990) Fitting ellipses and predictingconfidence envelopes using a bias correctedKalman filter, Image and Vision Computing,Vol 8, No 1, 37-41.

Figure 6.

36