SOLVING GEOMETRIC CONSTRAINTS IN A PARALLEL NETWORK

Robert B. FisherMark J. L. Orr

Department of Artificial IntelligenceEdinburgh University

ABSTRACT

We describe a network implementation ofthe SUP-INF method of solving sets of inequali-ties that has advantages over previous implemen-tations. The cost of symbolic manipulation istransferred to compile-time allowing speed up atrun-time due to parallel evaluation. Further,allowing iteration in the network improves thecompetence of the method when working withnon-linear expressions. We use the network toimplement a geometric reasoner for a computervision program and show that it meets the gen-eral requirements for such a system.

1. INTRODUCTION

In a previous paper (Orr 1987a) we investi-gated the general nature of a geometric reasonerfor computer vision, concerned more withspecification than implementation. Amongst theconclusions we made, was that algebraic inequali-ties were a suitable representation for constraints,because it was a uniform mechanism for a varietyof model-data relationships, it represents both apriori and observed relationships and easilyallows model variation. Constraints generallyinvolve unknown quantities not measured fromthe image or specified in the models and forwhich estimates are sought. To find estimates itis necessary to combine related constraints andthis involves solving sets of algebraic inequalities.

This approach has already been demon-strated by Brooks (Brooks 1981). A crucial partof his constraint solving mechanism was a sym-bolic manipulator for algebraic expressions. Thishad drawbacks of a high cost for symbolic pro-cessing and an inability to properly handle non-linear constraints.

AcknowledgementsThis work was performed under Alvey GrantGR/D/1740.3.

Section 2 describes an implementation of theSUP-INF method that does away with the needfor symbolic manipulation at run-time, has anatural parallel structure and copes better withnon-linear constraints. In section 3 we show howthis implementation can be used for geometricreasoning and how it meets the specifications out-lined in our previous paper (Orr 1987a).

2. SOLVING CONSTRAINTS IN PARALLEL

The basic constraint solving method we useis Bledsoe's SUP-INF algorithm (Bledsoe 1975),later refined by Shostak (Shostak 1977) andBrooks (Brooks 1981). Constraints are expressedin the form:

or

where the x. are members of a set {x... x~....x }of variables and f j and g. are values or expres-sions involving some or aA of the x.. A solutionof the constraints would be a substitution of realvalues for the variables that maintained the truthof each inequality. The goal of the algorithm, fora given set of constraints, is:

(1) to decide whether the set of possible solu-tions is empty,

(2) to find bounds on the value that a givenexpression (involving some or all of the x.)can attain over a non-empty set of solu-tions.

The algorithm is based on the recursiveapplication of the functions SUP and INF on theexpression to be bound and its sub-expressions.SUP returns an upper bound (supremuxn) andINF a lower bound (infimum). In Brooks' (Brooks1981) program the simplification of constraintsand the application of SUP and INF was handledby symbolic manipulation at run time. Wepresent a new implementation of the SUP-INFmethod that has the following advantages:

87

(1) it transfers the cost of symbolic manipula-tion from run-time to compile-time.

(2) it improves the performance of the algo-rithm for non- linear constraints,

(3) it has a natural parallel structure.The implementation has the structure of a

network with nodes and connections as describedbelow.

Structure of the network

The network consists of two types of nodes:value nodes and operation nodes. The valuenodes acquire numerical SUP and INF bounds ontheir associated algebraic variable or expression.The bounds are computed from connections withother value nodes or with operation nodes thatreceive inputs from other value or operationnodes. Each time new bounds are computed thechange propagates over the network causing othernodes to acquire new bounds. The changesbecome smaller as the bounds get closer and thenetwork converges asymptotically to a stablestate when the desired bounds on variables orexpressions of interest can be extracted from theassociated value nodes.

Network CreationA network is constructed by linking

together several network fragments or modules.Each module represents a particular instance of acommon constraint type and there may be morethan one module of the same type in the net-work. The structure of modules is defined by anoff-line compilation process. Consequently, theon-line program which uses the network, such asthe geometric reasoner described in section 3, onlyhas to connect instances of the appropriatemodules to solve the problem at hand.

A module is compiled from a list of alge-braic inequalities such as:

+ 2

The inequalities are written by a humanprogrammer after due consideration of the 'prob-lem' that the module 'solves'. An example fromgeometric reasoning (section 3) is the problem ofrelating two pairs of direction vectors with arotation when the relation between the pairedvectors is that one is the rotation of the other.The relations between all the (scalar) variablesoccurring in the problem are expressed as inequal-ities. These give both exact relationships and'heuristic' bounds. If an equality is encountered

then it is split into two inequalities:x - expr becomes:

x ^expr & x ^expr

If a product is encountered, then it is splitinto four inequalities involving the signedreciprocal Csrecip") function:

x * y ^ z becomes:

x < z • srecip(y)

y < z • srecip(x)

x ^ -z * srecip(-y)

y ^ -z • srecip(-x)This function has the definition:

srecip(x) - if x > 0 then 1/xelse 'undefined'

and consequently has the effect of turning off andon constraints according to the sign of its argu-ment.

Recursive constraints are allowed such as:

which becomes:

x > (1 - y 2 ) * srecip(x)

x < ( y 2 - l ) * s r e c i p ( - x )but are treated differently by boundsimplification (see below).

Symbolic manipulationBefore compiling the network, the list of

inequalities is checked for correct syntax,simplified and processed by the functions SUPand INF. In general this is a hard problem butthe constraint manipulation system (CMS) ofBrooks' program ACRONYM (Brooks 1981) atleast provides some competence. We haveextended this CMS to cope with square roots,powers of variables, symbolic rather thannumeric bounds on products where appropriate,the unsigned reciprocal function and theundefined value (Fisher 1987b).

Simplification is only applied to non-recursive constraints where the variable on theleft hand side of the inequality does not appearanywhere in the right hand side. Recursive con-straints are difficult to handle and usually get

reduced to the trivial:-infinity Kx <+infinity

The CMS could be used directly (as inACRONYM) by the on-line program. Measure-ments made by the program would add new con-straints providing more scope for simplificationand eventually to bounds on variables andexpressions that are not measured directly. How-ever, symbolic reasoning is computationallyexpensive and not suited to wide scale parallel-ism.

A more compelling reason for using a net-work is that it can iterate to better bounds overnon-linear constraints than the single passmethod of the CMS. Consider the followingexample.

x < 1 + 1/y

y > 1 + 1/x

0.1 10

0.1 < x <10

The CMS (somewhat simplified) finds:SUKx) - 1 + l/INF(y)- 1 + 1/(1 + l/SUP(x))

When it gets to the embedded SUP(x) ituses the numerical bound 10 to produce:

SUKx) - 1 + 1/(1 + 1/10)-1 .91

However the network computation iteratesto the (analytically) best bound:

SUP(x) - 1.62

Network CompilationValue nodes are created for all variables

occurring in the constraint list. These are con-nected by various operator nodes that extractvalues from value nodes or other operators. Theconnections are determined by the expressionsfound in the constraints. The following is a listof the actions taken by the compiler when itencounters the specified expression type:

constant:An operation node (with no inputs) iscreated that supplies the given constant.

variable:An operation node is created that extracts

the SUP (or INF) of the associated valuenode.

plus: An operation node is created that adds theresults of the recursively compiled sub-expressions.

max (or min):SUP(max(list)) is compiled to bemax(SUP(list)) (analogously for INF and'min'). Thus subfragments for each sub-expression in the list are created and linkedto a series of connected binary 'max' (or'min') nodes. Network evaluation isdifferent for max (or min) nodes createdfrom SUP or INF in their use of defaultswhen not all arguments are evaluated(which may arise from timing delays oralternative expressions being undefined).The INF max function returns a value if atleast one argument is evaluated; the SUPmax function only returns a value when allarguments are evaluated.

times:SUP(A*B) is expanded to:

max(INF(A)*INF(B).INF(A)*SUP(B),SUP(A)*INF(B).SUP(A)*SUP(B))

and then compiled. The same for INF(A*B)except 'max" is replaced by 'min'.

redp(E) (where E is an expression):A test-case node is required for the recipro-cal function. Test-case nodes select theiroutput according to a test defined atcompile-time and carried out at run-time. IfSUP is the desired bound, the test-case con-struction is:

if INF(E)>0 or SUP(E)O

then 1/INF(E)

else plus_infinityIf INF is the desired bound then:

if INF(E)>0 or SUP(E)<0

then 1/SUP(E)

else minus_infinity

sredp(E) (where E is an expression):This is the signed reciprocal function where:

89

srecipd) - if x > 0 then 1/xelse 'undefined'

If SUP is the desired bound, a test-case nodeis created selecting:

if INF(E)X)

then 1/INF(E)

else 'undefined'If INF is the desired bound then the test-case construction is:

if INF(E) >O

then 1/SUKE)

else 'undefined'vn (where v is a variable and n is odd):

A sequence of 'times' operation nodes arecreated and linked to the SUP (or INF) ofthe variable. The output of each 'times'operation becomes the input to the next.

vn {where v is a variable and n is even):If SUP is the desired bound then sequencesof 'times' nodes are created and linked toboth the INF and SUP of the variable and afinal 'max' node linked to the output of eachsequence. If INF is the desired bound then a'test-case' node is created selecting:

if SUKv) < 0

then [SUKv)]n

else if INF(v) > 0

then [lNF(v)]n

elseOsquare_root(E) (where E is an expression):

The positive square root is assumed. If SUPis the desired bound then:

if SUP(E) > 0

then sqrt(SUKE))

else 'undefined'If INF is the desired bound:

i fINF(E)>0

then sqrt(INF(E))

else 'undefined'

As the same expressions may be used morethan once in different constraints in the samemodule, the recursive compiler uses a previouscompilation for a expression if one exists, thusavoiding duplication. Another simplification isthe reduction of multiple constraints to a single'min' or 'max' function:

, v - — becomes:

v ^min(Ej, Ey. ...)A similar simplification is performed for lowerbounds using the 'max' function.

To illustrate the creation of a networkmodule, suppose we are interested in the 'prob-lem':

A < B - Cwhich entails the further constraints:

C < B - A

This list of constraints would be the inputto the CMS (Fisher 1987b). that would have lit-tle to simplify but would recursively apply theSUP and INF functions symbolically to find:

SUP(A) - SUP(B) - INF(C)

INF(B) - INF(A) + INF(C)

SUP(C) - SUKB) - INF(A)The compiler then produces the network

shown in figure 1. This is a trivial example thateven fails to compute both bounds on the param-eters involved. In practice (see section 3)modules are larger and more complicated.

ModularisationThe run-time program constructs and evalu-

ates its own networks according to the problemsit is presented with. We assume that problemscan be broken down into several parts each ofwhich can be managed by an instance of somepreviously compiled module. Suppose we havethe following two constraints:

90

Figure 1: the network for A < B - C.

- z

y - wA network for this problem would be con-

structed out of two instances of the moduledefined above for the constraint type:

A < B - Cand connected as shown in figure 2. The modulescan be thought of as black boxes with connectionsto the outside world. For the first constraint theconnections A->x, B->y and C->z are made,while for the second constraint A->y, B->z andC->w.

Network EvaluationThe values at each node are computed using

the values at the connecting nodes. The SUP(INF) computation chooses the minimum (max-imum) of each of its current bounds and itscurrent value. Including the current value in thecalculation ensures that bounds can only gettighter. Thus if:

SUP(A) < a r SUP(A) <&2. ...

then:

SUP(At+1) - min(SUP(At). a r a^. ...)is the updating function for the supremum of Afrom time t to time t+1.

The networks of modules are designed to beevaluated in parallel. The whole network could

Figure 2: two connected modules.

be evaluated synchronously or asynchronously ina MIMD processor with non-local connectivity.Ideally, each node would be stored in a separateprocessor, continually polling its inputs andupdating its output if appropriate.

So far we only simulate the network seri-ally. When the change at a node drops below apreset threshold its dependent nodes no longerrequire re-evaluation. When no more nodes needto be evaluated, the network has reached a stablestate and processing can stop. The other way ofstopping the network is the detection of an incon-sistency signaled by a pair of bounds crossingover (the SUP of some value node becomes lowerthan its INF).

3. GEOMETRIC REASONING FOR COMPUTERVISION

We have previously (Orr 1987a) character-ised the tasks carried out by a geometric reasonerfor computer vision as a small set of functionsthat operate on and return certain types ofobjects, amongst which was the type 'position'.The five primitive functions needed are:LOCATE:

deduces position constraints from pairingsof model features to image features,

MERGE:integrates separate position estimates,

TRANSFORM:transforms one position by another (ie.given B relative to A and C relative to B find

91

C relative to A).

INVERSE:transforms the position of A relative to Binto B relative to A.

PREDICT:deduces what data should be present in theimage, given a position and a model feature.We also noted that the position data type

was required to represent uncertainty because ofthe inherent errors of image measurements.

That paper only addressed the question ofwhat must a geometric reasoner do. leaving openthe question of how it does it. We describedabove an implementation based on networks ofalgebraic constraints. In this section we willdescribe specific networks that can be used forgeometric reasoning and show how the they meetthe requirements of our previous paper.

Underlying our implementation is the use ofalgebraic constraints to represent knowledgeabout the world. For example, to represent a posi-tion requires specifying upper and lower boundson each of the six degrees of freedom. Thissatisfies the requirement of representing uncer-tainty because the true value of each parametercan lie in a range.

The TRANSFORM function is implementedas a network module. Looked at as a black box.the TRANSFORM module has three sets of portsto the outside world representing three positions(18 parameters in total): the position beingtransformed, the transforming position and theresulting position. When operating in the contextof an evaluating network, if any two of the setsof ports receive bounds from outside, the modulewill reflect the new situation by setting newbounds on the third set of ports. The INVERSEfunction is implicit in the network through thebi-directionality of all modules involving posi-tions.

The MERGE function is carried out at thenodes linking the ports from different modules.Each port is 'saying something' about the boundson some variable and if two or more ports arelinked then they either agree (the bounds inter-sect) or disagree. In the latter case, an incon-sistency has been detected - precisely what theMERGE function was designed to do. Further,the intersection of the bounds also improves theestimates of the variables.

The function LOCATE is implemented by aseries of network modules, one for each general

type of constraint derivable from the possiblepairings between model features and imagefeatures. These features vary from one visionsystem to the next and the corresponding con-straints will also differ. In Brooks' ACRONYM(Brooks 1981) the models were 3D volumetricprimitives while the image features were 2D. Ourown system. IMAGINE (Fisher 1986). is based on3D models augmented by 2D viewpoint informa-tion (Fisher 1987a) and 3D stereo images.

In our case, most pairings between a modeland an image feature can be reduced to a set ofpairings between 3D direction and location vec-tors. The general constraint types are then dis-tinguished by the numbers of location and direc-tion pairings involved. Suppose we have the twopairings:(1) a straight model line with an image edge,

and(2) a conical model surface with a similar image

surfaceEach of these can be broken down into the

pairing of a direction vector and a point locationin the model with a direction and location in theimage. Even though the features involved arecompletely different, they both present the samegeneral type of constraint and therefore can be'LOCATEd' by the same type of network module.

Part of our vision system is a catalogue ofconstraints (Orr 1987b) available from pairingsbetween feature types in our modeling schemeand their corresponding image feature types.Whenever the geometric reasoner encounters anew hypothesis (feature pairing) it can look upthe type of pairing in the catalogue and find out:

(1) which data vectors to pair with whichmodel vectors, and

(2) which network modules to use.This enables it to create new instances of the

modules and link them into the already existingand evaluated network. This network reflectsconstraints already integrated and may containmany modules created from several previoushypotheses. If the network was previously in aconsistent state but, after the introduction of thenew module, becomes inconsistent then the rea-soner can deduce that something was amiss withthe new hypothesis. Otherwise, the new modulemay help to decrease the uncertainty in the net-work by pushing some bounds closer together.Thus the geometric reasoner fulfills its role ofaiding image interpretation.

92

Every module implementing a LOCATEfunction (where the input is from the image andmodels and the output is a position constraint)also implements a PREDICT function (where theinput is from a position constraint and themodels and the output is a constraint on some-thing in the image). There is a symmetry betweenthe two functions because the network moduledoes not distinguish which of its ports are forinput and which are for output. It simply main-tains a set of relations between the values currentat each of its ports.

Analysis of the geometric constraints usedso far has shown repeated patterns in their alge-braic structure and hence that the constraints canbe constructed via the composition of six primi-tive structures:

(1) I Sj - $2 I ^ threshold — two sealers areclose

(2) I Vj - Vj I < threshold — two direction vec-tors are close

(3) II L - L, I ^ threshold — two points are close

(4) Pj(P2^ ™ P3 ~ * position transformed by aposition gives a position

(5) Ky^) - y_2 and PCy )̂ - v^ — a pair ofdirection vectors transformed by a positiongives a pair of direction vectors (both rela-tions need not be used, but are necessary forcompletely deducing the position from thepaired vectors)

(6) POp - Lj. PQg) - L and PQj) - 1̂ - threepoints transformed oy a position give threepoints (again, only one or two points maybe used)

Thus, the constraint that says a transformedmodel direction vector (m) must lie within a dis-tance (t) of an observed data vector (dj

I P ( m ) - d J < tcan be decomposed into instances of modules 2and 5 connected.

The network, modules are defined in termsof the algebraic relationships between the inter-face variables. When compiled, they mainly con-sist of operation nodes. The sizes of the sixmodules (in operation nodes) are:

module 1: 26 nodesmodule 2: 826 nodesmodule 3: 297 nodes

module 4: 2381 nodesmodule 5: 1589 nodesmodule 6: 1225 nodes

the following model direction vectorsm, - (-0.51.0.83,0.22)m i - (0.68,-0.23.0.69)

are rotated rigidly by position P to give the vec-tors P(jSj) and Kfflj). Then, assume we observetwo data vectors

d« - (-0.40,0.91.0.04)j ^ - (-0.52.-0.67,0.51)

that are paired with m« and mj respectively.(This pairing results from the model-based sceneanalysis. An example pairing might arise fromusing a 3D orientation discontinuity and the nor-mal of a planar surface patch.)

The pairings are represented using aninstance of module 5 from above, linked to them. and d. vectors. Evaluating this simple net-work, the following bounds are achieved on therotation (represented as a quaternion):

Low (0.729.0.249.-0.622.-0.141)High (0.731.0.252.-0.618.-0.139)

where the true value is(0.73,0.25.-0.62.-0.14)

The result required 46 network update cycleswith 3919 operation node evaluations, where allevaluations in each cycle could be executed inparallel. This gives the time necessary for valuesto completely propagate through the layers ofsimple function units several times before con-vergence.

Now. suppose instead we anticipate that thevectors d, are displaced from their true positionby some observational error of maximum magni-tude E. We then know

We conclude this section with an example ofestimating an object's 3D orientation. Assume

I P C t t ) ^ ! <Eso we can now use two instances of module two.

Evaluating this network with E - 0.05 pro-duces the new bounds on the estimated rotation:

Low (0.615,0.149,-0.810.-0.220)High (0.804.0.370.-0.438.-0.076)

which contains the correct rotation given above(63 network update cycles involving 7591 nodeevaluations).

An alternative to using module two entailssetting the bounds on the d- directly, whichallows for non-isotropic errors. More exact errorrelationships could be represented algebraicallyand linked by other modules.

If we make a third model-to-data vectorpairing, then we connect new instances ofmodules two and five. This final network isshown in figure 3. Suppose this new vector isobserved with E - 0.02. Then, the new estimatedrotation is:

Low (0.652,0.173.-0.805,-0.194)

93

High (0.799.0.341,-0.464,-0.097)which has reduced variance (52 network updatecycles involving 12360 node evaluations). Theaverage estimated parameter value is:

(0.725.0.257.-0.634,-0.145)which compares favourably with the true value.Note that the bounds give the full range ofallowable variation, instead of a statistical esti-mate.

This example demonstrates combining solu-tions to several constraint problems to fully con-strain a larger problem. In practise, as more evi-dence is obtained, more network modules can beconnected to further refine parameter estimates,especially as the pre-defined modules are usablefor a variety of geometric relationships.

4. RELATED WORKThe use of algebraic inequalities to represent

geometric constraints derives from Brooks'ACRONYM (Brooks 1981). as does the symbolicconstraint manipulation methods. The networkcomputation is similar to the many relaxation orconstraint satisfaction algorithms that are suit-able for parallel processing. However, it differsfrom the relaxation algorithms in that it is not aprobabilistic labelling computation and from con-straint satisfaction in that there is reduction of aninfinite continuous range of values rather thanselection from a finite set of discrete values.While the network relies on connections between

Figure 3: object orientation from three pairsof direction vectors

units, the computation is not in the distributedconnectionist form where the results areexpressed as states of the network. Instead, theresults are the values current at selected proces-sors.

The work presented here differs significantlyfrom two other network based geometric reason-ing systems. Hinton and Lang (1985) learnedand deduced positions of 2D patterns using a dis-tributed connectionist network, whose intermedi-ate nodes represented object position and gatedconnections between iconic image and modelrepresentations. Ballard and Tanaka (1985)demonstrated a 3D reasoning network whosenodes represent instances of parameter values andwhose connections represent consistency accord-ing to model-determined algebraic relationships.In both cases, patterns of network activity result,with the dominant pattern accepted as the answer(unlike here, where the result is explicit). Bothsystems also simultaneously select a model,which is treated separately in our analysis.

5. SUMMARYThe methodology we have investigated is

summarised here. We start with sets of algebraicconstraints associated with particular geometricrelationships (grouped into 6 primitive classes ofrelationships). Image observables are representedby variables at this stage. These constraints arethen processed by a CMS to produce symbolicbounds on each variable. The bounds are com-piled into a network where the structure of thenetwork reflects the structure of the expressionsfor the bounds. When observable variables getbound to measured values the other variables(position or model parameters) are forced intoconsistency by evaluating the network, which isdefined in a form suitable for parallel evaluation.Defining the constraints and applying the CMScan be done in advance. Then, at run time, net-works solving particular problems can be built byconnecting instances of the compiled constraintmodules, according to the structure of the prob-lem.

ACRONYMS CMS was optimal when pro-ducing numerical bounds on single variables oversets of linear constraints. Since we reproduce thesymbolic reasoning in the network, only substi-tuting data values later, the network must havethe same performance over linear constraint sets.Here, as the geometric constraints are mostlynon-linear, we cannot expect optimality. but ourextensions to the CMS and iterative evaluation in

94

the network improve the performance.While the network structure is capable of

implementing many computations, we have usedit for geometric reasoning, showing how the stan-dard reasoning functions and the use of 3Dmodels and 3D images require the definition ofvarious network modules. We demonstrated useof the network to support geometric reasoningwith an example of estimating an object's 3Dorientation.

REFERENCES1 Ballard. D. and Tanaka, H.. "Transforma-

tional Form Perception in 3D: Constraints.Algorithms and Implementation", Proc. 9thInt. Joint Conf. on Artif. Intel., pp 964-968,1985.

2 Bledsoe, W.W., "A new method for provingcertain Presburger formulas", Proc. 4th Int.Joint Conf. on Artif. Intel.. ppl5-21. 1975.

3 Brooks, R.A., "Symbolic reasoning among3-D models and 2-D images", Artif. Intel.,17. pp285-348, 1981.

4 Fisher, R.B., "From surfaces to objects:recognising objects using surface informationand object models", PhD thesis. Universityof Edinburgh. 1986.

5 Fisher. R.B., "SMS: a suggestive modelingsystem for object recognition". Image andVision Comp., 5. pp98-104. 1987a.

6 Fisher, R.B.. "A PROLOG version ofACRONYM'S CMS", Dept. of Artif. Intel.,working paper •**, Edinburgh University,1987b.

7 Hinton, G. and Lang, K., "Shape Recognitionand Illusory Conjunctions", Proc. 9th Int.Joint Conf. on Artif. Intel., pp 252-259,1985.

8 Orr, M.J.L., "Geometric reasoning for com-puter vision", to appear in Image and VisionComp., August, 1987a.

9 Orr, M.JL, "Geometric constraints in 3Dcomputer vision". Dept. of Artif. Intel.,working paper ***, Edinburgh University,1987b.

10 Shostak. R.E.. "On the SUP-EMF method forproving Presburger formulas*. J. Assoc.Comp. Mach., 24. pp529-543. 1977.

95