Multiple graph matching with Bayesian inference

Ž .Pattern Recognition Letters 18 1997 1275–1281

Multiple graph matching with Bayesian inference

Mark L. Williams a, Richard C. Wilson b, Edwin R. Hancock b,)

a Defence Research Agency, St Andrews Road, MalÕern, Worcestershire, WR14 3PS, UKb Department of Computer Science, UniÕersity of York, York, Y01 5DD, UK

Abstract

This paper describes the development of a Bayesian framework for multiple graph matching. The study is motivated bythe plethora of multi-sensor fusion problems which can be abstracted as multiple graph matching tasks. The study uses as itsstarting point the Bayesian consistency measure recently developed by Wilson and Hancock. Hitherto, the consistencymeasure has been used exclusively in the matching of graph-pairs. In the multiple graph matching study reported in thispaper, we use the Bayesian framework to construct an inference matrix which can be used to gauge the mutual consistencyof multiple graph-matches. The multiple graph-matching process is realised as an iterative discrete relaxation process whichaims to maximise the elements of the inference matrix. We experiment with our multiple graph matching process using anapplication vehicle furnished by the matching of aerial imagery. Here we are concerned with the simultaneous fusion ofoptical, infra-red and synthetic aperture radar images in the presence of digital map data. q 1997 Elsevier Science B.V.

Keywords: Graph matching; Bayesian inference; Relational consistency; multiple-graphs; Aerial images; Sensor fusion

1. Introduction

Graph matching is a critical image interpretationŽtool for high and intermediate level vision Shapiro

and Haralick, 1985, 1981; Sanfeliu and Fu, 1983;.Simic, 1991 . In essence the technique allows sym-

bolic image abstractions to be matched against one-another. It has been widely exploited in applicationssuch as stereo matching and relational model match-ing. In fact, it was Barrow and Popplestone who firstdemonstrated the practicality of relational graphs as

Ža flexible image representation Barrow and Popple-.stone, 1971 . More recently, the methodological ba-

sis has attracted renewed interest in two distinctareas. The first of these addresses the issue of how to

) Corresponding author. E-mail: [email protected].

compute a reliable distance measure between cor-rupted relational descriptions. Contributions here in-clude the use of binary attribute relations by Kittler

Ž .et al. 1993 , the efficient structural hashing idea ofŽ .Messmer and Bunke 1994 , and, Wilson and Han-

Žcock’s use of both constraint filtering Wilson et al.,. Ž .1995 and graph editing Wilson and Hancock, 1995

operations to control structural corruption. The sec-ond issue concerns the efficient search for optimalmatches. Recent efforts under this heading includethe use of mean-field annealing by Suganathan et al.Ž .1995 , the soft-assign technique of Gold and Ran-

Žgarajan Gold et al., 1996; Gold and Rangarajan,. Ž1996 and Cross et al.’s Cross et al., 1996; Cross

.and Hancock, 1995 use of genetic search.Despite this recent activity, the advances in

methodology have been confined almost exclusivelyto the matching of graph pairs. This is an important

0167-8655r97r$17.00 q 1997 Elsevier Science B.V. All rights reserved.Ž .PII S0167-8655 97 00117-7

( )M.L. Williams et al.rPattern Recognition Letters 18 1997 1275–12811276

omission. Multi-sensor fusion tasks pose an impor-tant practical challenge in applications such as re-mote-sensing and robotics, and are naturally posed as

Žmultiple graph matching problems Tang and Lee,.1992a,b . However, although principled multi-sensor

fusion algorithms are to hand, these invariably posethe matching process as one of feature registration or

Ž .clustering Hathaway et al., 1996 rather than rela-tional graph-matching. This means that the bulk ofavailable algorithms are effectively low-level andnon-contextual. Relational matching algorithms offerthe advantage of drawing on structural constraints inthe matching process. This means that they do notfor instance need to draw on an explicit transforma-tional model between the different sensors beingdeployed in the scene recovery task. In other words,they are calibration-free.

The aim in this paper is to extend our existingmethodology to the case of matching multiple-graphs.We have already addressed to dual issues of mod-

Želling relational distance Wilson et al., 1995; Wil-.son and Hancock, 1995, 1996 and efficiently locat-

Žing optimal matches Cross et al., 1995, 1996; Cross.and Hancock, 1995; Finch et al., 1997 . In particular,

our framework delivers the probability distributionfor pairwise matches using a Bayesian model ofmatching errors. The key to extending this frame-work to the case of multiple graphs is a means ofassessing relational similarity beyond the level ofgraph-pairs. We meet this goal by measuring thecompositional consistency of the pairwise matches.This process draws on a triangle of inference. If nodeA in the graph indexed l matches onto the node B inthe graph indexed m and this node in turn matchesonto the node C in the graph indexed n, then forcompositional consistency we make the inferencethat the node A in graph l must also match onto thenode C in graph n. Rather than imposing a series ofhard tests we assess the degree of compositionalconsistency using a Bayesian inference matrix. Mul-tiple graph matches are sought so as to maximise theelements of this matrix.

2. Theory

The basic task confronting us is to find the pair-wise matches between a set of N relational graphs.

Suppose that the graph indexed l is denoted byŽ .G s V , E where V is the set of nodes and E isl l l l l

the set of edges. The complete set of graphs is� 4denoted by Gs G N ls1, . . . , N . The pairwisel

match between the graph indexed l and the graphindexed m is represented by the function f :V ™Vl,m l m

from the nodes of the graph G to the nodes of thel

graph G .m

In our previous work we showed how the consis-tency of match between graph pairs could be gaugedusing the probability distribution for matching errorsŽWilson et al., 1995; Wilson and Hancock, 1995;

.Wilson and Hancock, 1996 . The distribution is de-rived from first principles and assumes that matchingerrors occur with a uniform probability P . Accord-e

ing to our model, the probability that the node a ingraph G matches onto the nodes b in graph G isl m

given by the following sum of exponential terms

K aP f a sb s exp yk H S, R .Ž . Ž .Ž . Ýl ,m e a< <Qb SgQ b

1Ž .

The physical quantity underpinning this distributionŽ .is the Hamming distance H S, R . This quantitya

counts the number of errors between the match� Ž . Ž . 4R s f b N a,b gE residing on the contextuala l

neighbourhood of the node a in graph G and eachl

of the set of feasible configurations SgQ that canb

be formed by permuting the nodes in the contextualneighbourhood of the node b from graph G . Them

Ž . < R a < ŽŽconstants K s 1 y P and k s ln 1 ya e e. Ž .P r P are regulated by the uniform probability ofe e

matching errors P .e

To extend our methodology from matching graphpairs to the case of multiple graph matching, werequire a means of augmenting our consistency mea-sure. In particular, we require a way ensuring thatthe system of pairwise matching functions are mutu-ally consistent over the multiple graphs. In develop-ing our new framework we appeal to the idea induc-ing inference triangles by forming compositions ofthe pairwise matching functions. In other words, we

Ž . Ž .aim to ensure that if f a sb and f b sc thenl,m m ,n

the composition results in the consistent matchŽ .f a sc. However, rather than imposing this tran-l,n

sitive closure condition as a hard constraint, we aimto capture it in a softened Bayesian manner. We

( )M.L. Williams et al.rPattern Recognition Letters 18 1997 1275–1281 1277

Ž .measure the consistency of the match f a sc byl,n

computing the fuzzy-inference matrix

I a,c sŽ . Ý Ýl ,nG gG ,m/l ,m/n bgVm m

P f a sb P f b sc . 2Ž . Ž . Ž .Ž . Ž .l ,m m ,n

The matrix elements effectively accumulate the con-sistency of match over the set of transitive matchingpaths which lead from the node a of the graph G tol

the node c of graph G . These paths are specified bynŽ .the set of intermediate matches f a sb residingl,m

on the nodes of the graph G . The matrix averagesm

over both the possible intermediate graphs and theset of possible matching configurations residing onthe intermediate graph.

To update the matches residing on the multiplegraphs, we select the configuration which maximisesthe relevant inference matrix element. In other words,

f a sargmax I a,c . 3Ž . Ž . Ž .l ,n cg V l ,nn

In our implementation, the multi-graph matchingprocess is realised by parallel deterministic updatesalong the lines adopted in our previous work on

Ž .discrete relaxation Wilson et al., 1995 . However,we acknowledge that this simplistic optimisationprocess is prone to convergence to local optima.Suffice to say that the work reported in this paper isaimed at proof-of-concept of the multiple graph-matching technique. In other words, our aim is todemonstrate the utility of the fuzzy inference matrix

Ž .I a,c . Studies aimed at improving the optimisa-l,n

tion process are in hand and will be reported in duecourse.

3. Experiments

There are two aspects of our evaluation of themultiple graph-matching method. The first of these isa study focussed on the matching of three differentmodalities of aerial imagery against a cartographicmap. The sensing modalities are optical, infra-redline-scan and synthetic aperture radar. The secondaspect of our study is concerned with assessing thesensitivity of the matching process on simulationdata. The aim here is to illustrate the effectiveness ofthe matching process on data with known ground-truth.

Fig. 1. Optical image.

3.1. Real-world data

We illustrate the effectiveness of our multiplegraph matching process on an application furnishedby remotely sensed data. We use four overlappingdata sets. The data under study consists of partiallyoverlapped images and a cartographic map of theimaged area. Fig. 1 shows an optical image of a ruralregion surrounding an airstrip. Fig. 2 shows an infra-red line scan image of the airstrip, containing less ofthe surrounding region. Fig. 3 is a synthetic aperture

Ž .radar SAR image of the airstrip and some of itssurroundings. Fig. 4 is the cartographic map of area.The matching problem uses only the rural region tothe left of the main runway in each image. It isimportant to stress that the runway is used only as a

Fig. 2. Infra-red line-scan image.


Fig. 3. Synthetic aperture radar image.

fiducial structure for registering the ground-truth. Itdoes not feature in any of our matching experiments.

The matching process is posed as one of findingcorrespondences between linear hedge-structures inthe four data-sets. Relational graphs are constructedby computing the constrained Delaunay graphs forthe relevant sets of line-segments. The graphs corre-sponding to the images in Figs. 1–4 are shown inFigs. 5–8. It is worth stressing the difficulties inregistering the graph by-eye. In other words, themultiple graph-matching problem used to evaluateour inference-process is a demanding one. The opti-cal image gave rise to a 133 node graph, there were68 nodes in the graph derived from the SAR data

Fig. 4. Cartographic data.

Fig. 5. Graph for the optical image.

and 63 nodes in the graph derived from the infra-redimage. The region of map selected gave rise to 103separate hedge sections each represented by a nodein the map graph. To give some idea of the difficultyof the multi-graph registration process, it is worthmaking some comments on the qualitative featuresof the images under match. The clarity of the opticalimage made it easy to relate its salient structure tothose in the map, and to a lesser extent the SAR andinfra-red images. The weakest link in the inferencechain is provided by the matching of the SAR andinfra-red images. These two images contain quitedifferent structure. More importantly, their potentialmatches in the optical image are relatively disjoint.In fact, ground-truthing the matches between theSAR image and the infra-red image proved to be adifficult. The label-error probabilities used in theexperimental evaluation are computed using theoverlap statistics for the four data-sets. Fig. 9 showsthe error probabilities used in the matching process.

Fig. 6. Graph for the infra-red line-scan image.


Fig. 7. Graph for the synthetic aperture radar image.

Because it represents the weakest link, we com-mence our discussion of the experimental results byconsidering the match of the SAR and infra-redimages. Out of the 68 nodes in the SAR graph 28had valid matches to nodes in the infra-red graph.Initially none of the SAR graph nodes matchedcorrectly to the nodes in the graph derived from theinfra-red image. The matching process using no in-ference was unable to recover any correct matches.When the inference process was included in thematching process the final match recovered all of thepossible correct matches.

Out of the 103 nodes in the map graph 83 hadvalid matches to nodes in the graph derived from theoptical image. Initially none of the map graph nodeswere correctly matched and conventional graph tograph matching was unable to improve on this. Byincluding the inference terms into the matching pro-cess a final match including 58 correct matches wasachieved.

Fig. 8. Graph for the cartographic data.

Fig. 9. Label error probabilities used in the aerial image matchingexperiments.

3.2. SensitiÕity study

The inference process described in Section 2 isapplicable to the simultaneous matching of an arbi-trary number of graphs. However, the simplest caseinvolves the matching of three graphs. This allows usto illustrate the performance advantages offered bythe inference process under varying noise conditions.

The data used in this study has been generatedfrom random point-sets. The Voronoi tesselations ofthe random point-sets are used to construct Delaunaygraphs. By adding a predetermined fraction of extra-neous nodes to the original point-sets we simulatethe effects of graph-corruption due to segmentationerrors. We base our study on 50 node graphs. Weattempt matches under corruption levels of 10%,20%, 30%, 40% and 50%. We have also includedinitialisation errors in our simulation experiments.Our study is aimed at demonstrating the improve-ments that can be realised if a third inference graphis used to supplement the matching of two poorlyinitialised graphs. In the case of the poorly initialisedgraphs only 10% of the matches are correct. In bothcases the number of correct matches to the inferencegraph is initially 50%. In other words, the aggregateinitialisation error associated with the inference routeis 25%.


Table 1Results of simulation study

Corruption 0 10 20 30 40 50Best possible 100 90 80 70 60 50Fraction initially correct 25 25 25 25 25 25No inference 96 17.4 10.4 8.6 9.6 8.8Fixed inference 100 58.6 39.6 27.6 15.4 7.8Updated inference path 100 77.8 45.2 26.0 10.8 9.6

The results of our simulation study are sum-marised in Table 1. Here we show a comparison ofdifferent multiple graph matching schemes undervarying corruption and fixed initialisation error. Eachentry in the table represents the average over tenexperiments. In the first three rows of the table welist the fractional corruption, the best achievablefraction of correct matches, together with the fractionof initialisation errors. In the fourth row we show theresults when pairwise graph matching is attemptedwithout inference. The fifth row shows the result ofincluding the third graph as an intermediate state inthe triangle of inference. In this case the matching tothe inference graph is kept fixed and only the matchesbetween the original graphs are updated. The resultsare consistently better than the pairwise match, pro-vided that the level of corruption does not exceed40%. Finally, the sixth row shows the result obtainedif we also update the matches to the inference graph.Although there are fluctuations from one corruptionlevel to another, the average performance obtainedwith updated inference is better than its static coun-terpart shown in the fifth row.

4. Conclusions

We have detailed a fuzzy inference process formultiple graph matching. The basic idea underpin-ning the method is to use the recently reportedBayesian consistency measure of Wilson and Han-cock to construct an inference matrix for multiplegraph matching. This matrix allows strong pairs ofgraph matches to enhance the accuracy of weakermatches through a process of inducing inferencetriangles. The utility of our new multi-graph infer-ence technique is demonstrated on the matching ofaerial images obtained in three different sensor

modalities against a digital map. Here the inferenceprocess is demonstrated to render the matching ofotherwise unmatchable infra-red and SAR imagesfeasible.

There are a number of ways in which this study isbeing extended. In the first instance, the optimisationprocess adopted in our present study is a determinis-

Ž .tic discrete relaxation procedure Wilson et al., 1995 .One of the possibilities for enhancing the optimisa-tion process is to adopt a stochastic search proce-dure. For instance in a recent paper we have shownhow genetic search can be used to improve theperformance of matching graph pairs.

Discussion

Caelli: There is a lot of work by Bunke and otherson trying to solve problems of multi-subgraphmatching where they summarize the communalitiesbetween graphs by trees of subgraphs. I wonderwhether this technique could also be applied to thegeneral problem of multi-subgraph matching in termsof deriving common subsets of graphs from suchBayesianrrelaxation methods.

Williams: By the twinkle in your eye as you stoodup, I could see you were going to ask a verypertinent question to which I would have no ideahow to answer. Yes, I expect so; you could do anexhaustive search and get the answer, or you coulduse some other method and get a suboptimal answer.It is a question of how much effort you want to putin and how much pay-back you get. And this seemeda very simple, cheap way.

Mardia: In the end, when you talked about fusion, Iwould have liked to see a fused image, or somethinglike that, which comes from your different modali-ties, rather than the probabilities which you haveshown.

Williams: That was simply to demonstrate the pa-rameters we need, at least some estimate of the levelof corruption. The ultimate question is in fact: how


many nodes from one image, how many featuresfrom one image correctly match up with the featuresfrom the other image.

Mardia: So you do not end up with a fused image?

Williams: Well, if you then physically translate oneimage over the other, so that the features that aresupposed to correspond, lie on top of each other, youwould have fused images, but if those images wereheavily distorted then it would not be possible toregister them all simultaneously without rubbersheeting the image itself. Yes, it does lead to a fusedimage, but the answers are given in terms of howmany of the individual features correctly registeragainst features in the other image.

References

Barrow, H.G., Popplestone, R.J., 1971. Relational descriptions inpicture processing. Machine Intell. 6.

Cross, A.D.J., Wilson, R.C., Hancock, E.R., 1995. Discrete relax-ation on a Boltzmann Machine. In: Proc. ICANN 95, pp.491–496.

Cross, A.D.J., Wilson, R.C., Hancock, E.R., 1996. Genetic searchfor structural matching. In: Proc. ECCV96, Lecture Notes inComputer Science, vol. 1064 Springer, Berlin, pp. 514–525.

Cross, A.D.J., Hancock, E.R., 1995. Relational Matching withstochastic optimisation. In: Proc. IEEE Computer Society In-ternat. Symp. on Computer Vision, pp. 365–370.

Finch, A.M., Wilson, R.C., Hancock, E.R., 1997. Softening dis-crete relaxation. In: Neural Information Processing Systems,vol. 9. MIT Press, Cambridge, MA.

Gold, S., Rangarajan, A., Mjolsness, E., 1996. Learning withpre-knowledge: Clustering with point and graph-matching dis-tance measures. Neural Comput. 8, 787–804.

Gold, S., Rangarajan, A., 1996. A graduated assignment algorithmfor graph matching. IEEE PAMI 18, 377–388.

Hathaway, R.J., Bezdek, J.C., Pedrycz, W., 1996. A parametricmodel for fusing heterogeneous fuzzy data. IEEE Trans. FuzzySystems 4, 270–281.

Kittler, J., Christmas, W.J., Petrou, M., 1993. Probabilistic relax-ation for matching problems in machine vision. In: Proc. 4thInternat. Conf. on Computer Vision, pp. 666–674.

Messmer, B.T., Bunke, H., 1994. Efficient error-tolerant subgraphŽ .isomorphism detection. In: Dori, D., Bruckstein, A. Eds. ,

Shape, Structure and Pattern Recognition, pp. 231–240.Sanfeliu, A., Fu, K.S., 1983. IEEE SMC 13, 353–362.Shapiro, L., Haralick, R.M., 1981. Structural description and

inexact matching. IEEE PAMI 3, 504–519.Shapiro, L., Haralick, R.M., 1985. A metric for comparing rela-

tional descriptions. IEEE PAMI 7, 90–94.Simic, P., 1991. Constrained nets for graph matching and other

quadratic assignment problems. Neural Comput. 3, 268–281.Suganathan, P.N., Teoh, E.K., Mital, D.P., 1995. Pattern recogni-

tion by graph matching using Potts MFT networks. PatternRecognition 28, 997–1009.

Tang, Y.C., Lee, C.S.G., 1992a. A geometric feature relationgraph formalism for consistent sensor fusion. IEEE SMC 22,115–129.

Tang, Y.C., Lee, C.S.G., 1992b. Optimal strategic recognition ofobjects based on candidate discriminating graph with coordi-nated sensors. IEEE SMC 22, 647–661.

Wilson, R.C., Evans, A.N., Hancock, E.R., 1995. Relationalmatching by discrete relaxation. Image and Vision Comput.13, 411–421.

Wilson, R.C., Hancock, E.R., 1995. Relational matching withdynamic graph structures. In: Proc. 5th Internat. Conf. onComputer Vision, pp. 450–456.

Wilson, R.C., Hancock, E.R., 1996. Gauging relational consis-tency and correcting structural errors. In: IEEE ComputerSociety Computer Vision and Pattern Recognition Conf., pp.47–54.

Documents

Multiple graph matching with Bayesian inference