Inexact graph matching for structural pattern recognition

Pattern Recognition Letters 1 (1983) 245-253 May 1983 North-Holland

Inexact graph recognition

matching for structural pattern

H. Bunke Lehrstuhl fffr Informatik 5 (Mustererkennung), Universitiit Erlangen-Niirnberg, F.R. Germany

G. A l l e r m a n n Mannesmann Forschungsinstitut, Duisburg, F.R. Germany

Received 25 January 1983; revised 10 March 1983

Abstract: This paper is concerned with the inexact matching of attributed, relational graphs for structural pattern recognition. The matching procedure is based on a state space search utilizing heuristic information. Some experimental results are reported.

Key words: Attributed graphs, inexact matching, heuristic search, structural pattern recognition, silhouette recognition.

1. Introduction

In the structural approach, patterns are characterized as consisting of sub-patterns among which various relations exist (Fu (1982), Pavlidis (1977)). There are several ways for pattern description and representation in structural pattern recognition. A basic way is by means of strings. Due to the two-dimensionality of pictorial patterns, however, a two-dimensional description for- malism seems to be more adequate for certain applications. Consequently, the use of trees for pattern representation has been proposed (Fu and Bhargava (1973)). A natural generalization of trees is graphs. Their use allows the description of com- plex structures in a straightforward way on a high, problem-oriented level.

Besides representation of single patterns, the description of pattern classes is crucial. A method which has gained much importance over the last decade is the use of formal grammars (Fu (1982)). Grammars are favourable when a large number of different pattern classes is involved. At the other hand, there exist cases where the use of a grammar is not feasible, for example, when the set of sample

patterns is too small for the inference of a representative grammar. In this case, the patterns in the sample set can directly be used as prototypes, representing a class each. For the classification and /or description of an unknown pattern, one can match the input pattern against the prototypes.

Pattern matching is equivalent to graph matching when the patterns in question are represented by graphs. There are several papers in literature addressing the domain of matching by finding a graph isomorphism, e.g. Corneil and Gottlieb (1970). In practical applications, however, there is noise and distortions causing variation in both the input graph and the prototype. Therefore, our problem turns into a problem of inexact matching. This means given an input graph ~o and a set of prototypes ~ . . . . . ~ , all possibly different from co, we want to find out which one of the prototypes most closely resembles the input graph. As a consequence, we have to define some measure of similarity on graphs. Several approaches to inexact graph matching have been proposed in literature, e.g. Cheng and Huang (1981), Sanfeliu (1980), Shapiro (1981), Tsai and Fu (1979), Tsai and

0167-8655/83/53.00 ~ 1983, Elsevier Science Publishers B.V. (North-Holland) 245

Volume 1, Number 4 PATTERN RECOGNITION LETTERS May 1983

Chang (1982). The approach proposed in this paper is closey related to the work of Tsai and Fu (1979) and Tsai and Chang (1982), i.e. their defini-

tion of relational graphs is adopted and matching

is based on a state space search. Following are dif-

ferences: inexact matching is generalized to ar-

bitrary graphs; an extension to attributed graphs is

proposed; the cost functions are defined in such a way that the graph distance fulfils the properties of a metric under certain conditions.

2. Basic definitions

Let V s and V s be a set of node and branch labels, respectively. Each symbol belonging to V N

or V B is of the form (s,x) where s is the syntactic

component and x = (xl . . . . . xn) is a semantic vector consisting of attribute values associated with s.

The following definition is due to Tsai and Fu (1979).

Definition 1. An a t t r i b u t e d re la t iona l graph over

V= V N U V s is a 4-tupel ta = (N, B, a, e) where (1) N is a finite, nonempty set of nodes,

(2) B c_ N x N is the set of branches,

(3) a : N - - , V N is a function for labeling the nodes,

(4) c :B-- . V B is a function for labeling the

edges.

An element (x, y ) e B is interpreted as a directed

edge from node x to node y. Nodes x and y are call- ed the bordering nodes of the edge. An unat- tributed relational graph results when the semantic vector is empty, i.e. n =0 , for each syntactic sym-

bol. Using attributes besides syntactic labels has been proposed by Tsai and Fu (1979). It is a generalization of the formal graph models used by Tsai and Chang (1982), resp. Cheng and Huang (1981).

In this paper, we are concerned with distortions on graphs. Any such distortion can be described in terms of basic t ransformations, namely deletion, insertion, and substitution of nodes and edges. There are costs associated with each basic transformation. We denote the costs for deleting (inserting) a node with syntactic label a e V N by

DELNODE(0") (INSNODE(0")). The costs for substituting a node with syntactic label a by a node with

syntactic label r are given by SUBNODE (e, r). Similarly, SUBBRANCH(a,r) are costs for branch

substitution. Defining costs for branch deletion and insertion is slightly more difficult since these

t ransformations are dependent on the bordering nodes. When a node is deleted then all its outgoing and incoming branches vanish, too. Conversely, when inserting a node, there are usually branches to be inserted, too, for properly embedding this node in the rest of the graph. As a consequence, we define DELBRANCH(O') and INSBRANCH(O') as costs for insertion and deletion of a branch with syntac-

tic label (7 while none of the bordering nodes is in-

serted or deleted, respectively. In addition, DELBRANCH'(0") and INSBRANCH'(O') are correspon-

ding costs for branch deletion and insertion when at least one of the bordering nodes has been

deleted or inserted, respectively. While the above considerations are addressed to

distortions on only syntactic symbols, one has to take into account also distortions on attributes.

For deletion and insertion, the costs for attributes are assumed to be included in the costs concerning

the syntactic symbol. In the case of node substitu-

tion, however, the total costs for the syntactic symbol and the semantic vector are given by

SUBNODE ((7, r ) + ~ gilxi--Yil i = I

where

s = (a, (x, . . . . . x , ) )

and

t = ( r , (y~ . . . . . yn) )

are the labels o f the nodes involved and gi are weighting factors. The costs for edge substitution are analogously defined.

The costs as well as the weighting factors gi are some means for the user to express a priori knowledge about the problem domain. Low costs should indicate high likelihood for a particular t ransformat ion and vice versa. Remark that costs depend only on labels and attributes, not on particular nodes or branches. Costs have to range between zero and infinite where infinitely high costs

2,16


are to be defined for impossible t ransformations

(with respect to a particular problem domain).

When an inexact match between graphs o~ and a~2 is established, three cases can occur: a node n ~ N ~ can be deleted, or a node n t ~N~ can be

substituted by a node n_, e N 2, or a node n 2 e N 2 can be inserted. Following is a formal definition for inexact matching taking into account these cases. Formally, we augment both N l and N 2 by the symbol $ for indicating node deletions and in-

sertions. A node in co L, resp. w 2, which is mapped to the symbol $ is deleted from ~ l , resp. inserted

in a~2.

Definition 2. Let

091=(Ni, Bt,l~l,et) and ~2=(N2, B2,I.ta, e2)

be attributed relational graphs. Furthermore, let

N I = N I U { $ } , .N'2=N2U{$}, $ ~ N I U N 2.

An inexact match between co~ and co 2 is a partial

function f: /Vl -* N2 where

(l) f ( $ ) ~ $. (2) n ~ n ' = f ( n ) ¢ f ( n ' ) for all n , n ' e N l and

f (n ) , f ( n ' ) ~ N 2 . (3) For each n2eN2, there exists an nl e~rl with

f ( n i ) = n 2 and for each n l ~ N l , there exists an

n 2 e ~ : 2 with f ( n l ) = n 2.

A node n l e N l with f ( n l ) = n z e N 2 is substituted by a node n 2 in co 2. A node n~eN~ with f - t ( n j ) = = $ is deleted while a node n 2 e N 2 with f - 1 (n2) = $ is inserted. The property that f is a partial function is only addressed to the element

$ e N1, i.e. when there are no nodes to be inserted

in w2 then f ( $ ) is undefined. Condition (3) means

that each node in w~ (~o2) must be either substituted or deleted (inserted). Notice that Definition 2 covers inexact matches of any type. Particularly, it contains, as a special case, error- correcting isomorphisms introduced by Tsai and

Fu (1979).

Definition 3. Let co I and tz) 2 be attributed relational graphs and f:.K: l -'/if/2 an inexact match. The costs of that match, cos t ( f ) , are obtained by summing up all costs arising f rom deletion, inser-

;7 T

× ×

tM t J2 :

7

o)

Fig. 1. Two graphs.

7

Q Q

b)

tion and substitution of nodes and branches.

Note that mapping nodes by means of f induces

a mapping of branches, too. A branch (n, n')~B~ is substituted when there exists a branch

( f (n) , f (n ' ) ) e B2;

a branch (n, n ' ) e B l is deleted when there is no branch

( f ( n ) , f ( n ' ) ) e B2;

a branch (n, n ' ) e B z is inserted when there is no branch

(.f - l ( n ) , f - l(n')) e Bl.

Example 1. Two graphs co I and co2 are shown in Figure la ,b. An inexact match between m~ and oo 2

is given by

Let

f 1 : 2 ~ 4 , 3 - - 5 , 1 - - $ .

DELNODE(CQ=2 for a e {a , r } ,

SUBNODE(a, r ) = SUBNODE(r, O') = 1,

DELBRANCH(X) = 2 , DELBRANCH '(X) = l ,

XNSaRANCH (x) = 2,

then c o s t ( f 1 ) = 2 + 1 + 1 + 1 =5 by deleting a node with label a , deleting two edges (where the costs are DELBRANCH'(X) in this case), and substituting one node label. Another inexact match is given, for example, by

f 2 : 1 - - 4 , 2 - - 5 , 3 - - $ .

Here, we have c o s t ( f z)= 6. There are no attributes in this example.

247


Definition 4. Let fl . . . . . fn be all inexact matches which are possible between an attributed relational graph co! and co_,. The optimal inexact match f * between col and co2 is defined by

c o s t ( f * ) = min {cost(f)}. fl . . . . . fn

Lemma 5. Let cost(cot, co 2) denote the costs of the optimal inexact match between col and co,.; cost(wt, co2) is a metric t i f the following holds:

(1) SUBNODE (O', O') = SUBBRANCH(O', O') = 0 for all syntactic symbols a, and SUBNODE(a, r), SUB- BRANCH(a, r ) > 0 for ere 1., and the costs for any deletion or insertion are greater than zero.

(2) SUBNODE(O', 1.) = SUBNODE(1., O'), DELNODE(O')

= INSNODE (O'), DELBRANCH'(O') = INSBRANCH'(O')

for any symbols er, 1.; the analogous conditions hold true for branches.

(3) Let c2 and c 3 be the costs for two basic transformations yielding the same result as one basic transformation with costs cl, then always

C l ~ C 2 + C 3 •

The proof is straightforward by independently concluding all metric properties from condition (1)-(3). An example for condition (3) is

SUBNODE (O', l") __< SUBNODE(O', ~O)

+ SUBNODE (Q, l ').

Notice that none of the approaches to inexact graph matching cited in the introduction is a metric.

3. Finding the optimal inexact match

Referring back to the problem stated in the introduction of finding that prototype col among ml . . . . . con which most closely resembles an input graph co, a straightforward solution is to deter-

1 This means (a) cost(w,, oaz) -> 0 and cost(oat, oa,) = 0 iff oat=w,; (b) cost(w, co,)=cost(oa2, w,. oat); (c) cost(oat, w3) -<cost(col, m2)+cost(oJ.,,cob) for any attributed relational graphs co t, co:. cob-

mine cost(co~, co) . . . . ,cost(con, co) and to pick up that prototype with minimum costs. However, since we have to take into consideration all inexact matches between co and each prototype for finding the optimal inexact match, high computational costs are involved. As a matter of fact, finding the optimal inexact match requires exponential time with respect to the number of nodes. In this section, we introduce heuristic search according to Nilsson (1981) for the purpose of effectively computing the optimal inexact match. The costs for deletion, insertion and substitution of nodes and edges are employed to guide the search. The method described here is closely related to the approach proposed in Tsai and Fu (1979), Tsai and Chang (1982),

Matching co~ against co2, a state S in the search space is given by a partial inexact match fs between co~ and co2. Thus a state can be represented through a subset S c_ ,,91 × N2 with

( x , y )~S ~* fs (x)=y,

where the mapping fs fulfils condition (1) and (2) in Definition 2. A state in the search space gives that part of the desired inexact match which has been found until a certain moment. The initial state is defined by S0=0. A final state Sf is characterized by covering both N l and N 2. Note that the inexact match corresponding to a final state fulfils also condition (3) in Definition 2.

By means of expansion of a state S, a new state S' is generated. Expansion of a state

s = { ( x ~ , y l ) . . . . . (xn, y.)}

is defined by adding a new pair (x, y) e.~l x -'V2 to S where

X ~ {X 1 . . . . . X n } - - {S}, y ~ {Yl . . . . . Y n } - - {S},

(x ,y)*($, $). Expansion is defined only for non- final states. The meaning of expanding a state S by the pair (x,y) is to augment the partial inexact m a t c h f s by one more element. As an example, the search space for matching co! against co2 in Exam- ple 1 is given in Figure 2. The initial state is the root of the tree while tip nodes are final states.

The costs of a state S are given by the costs of the partial match fs according to Definition 3. The costs of the initial state are defined as being equal

248


to zero. Now, the problem of finding an optimal

inexact match is equivalent to finding a final state with minimum costs among all other final states. A solution to this problem is provided by starting with the initial state and expanding iteratively that state with minimum costs. Expanding is repeated as long as the state with minimum costs is no goal state. According to Nilsson (1980), this procedure is guaranteed to yield the optimal inexact match. As an example, the costs corresponding to the states are given inside the circles in Figure 2. The optimal inexact match between co~ and co_, is given by ./'1 in Example 1.

For speeding up the search, additional heuristic information is to be utilized. According to Nilsson (1980), we use an estimate

cost (S) = cos h (S) + cost2(S)

for the costs at state S where costl(S) is the costs of the partial match fs as defined beforehand, and cost2(S) is a lower bound estimate of the costs which are expected when the rest of ~1 and oJ 2 not yet taken into account at state S will be matched against each other. It can be formally proven that with such an estimation fewer nodes will be ex- panded, compared with a search utilizing no

heuristic information, and the algorithm is still guaranteed to find the optimal inexact match.

For the computation of cost2(S), suppose that subgraph al of co I has already been matched against subgraph cq of co2. Now the objective is to get an optimistic assessment of the future costs, cost2(S), when matching the remaining part ¢ o l - a ~ against co , -0t 2. For this purpose, we focus on subgraphs ,6' 1 and ,82 each node of which is connected via at least one edge with st and et2, respectively. We divide cost2(S) into one part A arising from insertions and deletions, and into another part B arising from substitutions, i.e. cost2(S) = A + B. Part A is calculated by determin- ing the difference between ,81 and ,g 2 with respect to the number of nodes. The difference serves as a basic for an optimistic assessment of future costs arising from insertions and /or deletions. Part B, i.e. future substitution costs, are optimistically assessed by matching in a tentative fashion independently in the cheapest way each node in ,81 against a node in ,sz. It can easily be seen that our procedure yields a consistent lower bound estimate for the real future costs. As the experiments described in the next section have shown, assessing the future costs results in a considerable speed up.

{(1,4)}

/

{(1,4),(2,5),(3,$) } @

{(1,4),(2,$

{(1,4),(2,$)} Q

{ ( I ,5)}

{(i,5),(2,4)} {(1,5),(2,$)}

{(1,5) ,(2,4),(3,$2)

{(I,$)} /

/

\ / / ',\ g

{(I,$) ,(2,4)} {(I,$)

{(I,$),(2,4),(3,~

(2,5)}

,(3,5)}~6~ {(I,5),(2,$),(3,~ {(I,$),(2,5),(3,4)}

Fig. 2. Search space for matching eJ I against w2 (see Figure 1). Costs are encircled.

249

Volume 1, N u m b e r 4 P A T T E R N R E C O G N I T I O N L E T T E R S M a y 1983

Table I

Syntac t ic symbol Mean ing

a a < 900

b a = 9 0 °

c 90 ° < g < 180 °

d 180 ° < a < 2 7 0 °

e a = 270 °

f 270 ° < ~ < 360 °

4. Experimental results and conclusions

The method described in Section 2 and 3 has been implemented in PASCAt on a TR 440 Com- puter at the University of Erlangen. Geometric silhouettes have been used for testing the programs and gaining experience. The transformation of silhouettes into graphs has been done manually.

Two different methods for silhouette representation by means of graphs have been used. In the first method, silhouette corners correspond to graph nodes while silhouette edges correspond to graph branches. Syntactic symbols 'a', 'b ' , . . . . ' f ' have been used for labeling nodes corresponding to corners with an angle a. The meaning of these labels is shown in Table 1. There is only one node attribute, namely the size of the angle in degrees. The syntactic symbol E is used for all graph branches while the (only) branch attribute defines the length of the corresponding silhouette edge. For an example, look at Figure 3a where a graph representation of Figure 4c is given. In the second representational method, silhouette corners correspond to graph edges while silhouette edges are represented through graph nodes. All labels are identical to

(c,i06)

()

(E ,9)

(), (a,74)

(E,5)

(E,~O)

(c,i06)

-()

(E,9)

( a , 7 4 )

aJ

(E ,5 )

(a,74) ~ ~ (a,74)

(E,t0)

b)

Fig. 3. G r a p h rep resen ta t ion of F igure 4¢ by means of two

methods .

method 1 up to the fact that there is an additional syntactic symbol 'g' indicating parallel silhouette edges having the distance of those edges as attribute. For an example, see Figure 3b which again represents Figure 4c.

1 1

3 3

a ) bJ

3

c)

Fig. 4. Five p ro to types .

dJ

3

e)

250

Volume I0 Number 4 PATTERN RECOGNITION LETTERS May 1983

Table

a b c d e f $

0 0.5 1.0 1.5 2.0 2.5 2.0

0.5 0 0.5 1.0 1.5 2.0 2.0

1.0 0.5 0 0.5 1.0 1.5 2.0

1.5 1.0 0.5 0 0.5 1.0 2.0

2.0 1.5 1.0 0.5 0 0.5 2.0

2.5 2.0 1.5 1.0 0.5 0 2.0

2.0 2.0 2.0 2.0 2.0 2.0 -

Table 3

Input pattern Best matching Costs Costs

prototype Method 1 Method 2

a a 1.8 3.0

b d 1.98 2.58

c c 1.36 1.36

d d 2.7 2.7

e c 7.25 7.85

The substitution cost matrix for matching graphs represented according to the first method is given in Table 2. The entry in column c and line d gives, for example, the costs for substituting node label d by c. The last column, resp. last line, gives deletion costs, resp. substitution costs. The weights gt . . . . . gn of Section 2 have been choosen as g~ = 10 -3, n = 1. Both the syntactic labels and attributes depend on the angle size. While the categorization into angle types according to Table 1 induces a coarse rating based on relatively high weights, a finer distinction within each category is obtained by means of the attributes. Remark that there are other ways feasible for selecting node labels (and costs), e.g. by choosing the same label for the case 0 < a < 90 ° and a = 90 °. Considering branches, only deletion and insertion costs, i.e. no substitution costs, are needed. We have used the following costs:

DELBRANCH(E) = INSBRANCH(E) = 1.5

and

INSBRANCH'(E) = DELBRANCH'(E) = 1.0.

The costs for representation method 2 are defined similarly. All costs have been heuristically determined.

1 r

Five prototypes and five input patterns are shown in Figure 4 and 5, respectively. The results of matching the input patterns against the prototypes are given in Table 3. The numbers at the edges of the input patterns refer to the correspon-

ding edges of the best matching prototype. Both representational methods yield the same results with respect to the best matching prototype and the edges, resp. corners, corresponding to each other. This isn't surprising since the role of nodes and edges is almost symmetric. Only the costs of the best match are different, in general. For a further example, look at Figure 6c where the prototypes in Figures 6a and b have been superimposed. Mat- ching the graph corresponding to Figure 6c with the union of both prototype graphs, we get costs of 16.00 (representation method 1) and 8.89 (representation method 2) for the optimal inexact match. Corresponding edges are indicated in Figure 6 by numbers 1 . . . . . 7, similiar to Figures 4 and 5.

So far, we have run a total of about 100 matches on our computer using additional prototypes and input graphs to Figures 4 to 6. It has turned out that it is not an 'art ' to determine suitable costs for basic transformations and the weightings gi (see Section 2). Instead, at least for our problem domain, the results of matching show stability with

a) b) c) d) e)

Fig. 5. Five input patterns.

251


o) b) c)

Fig. 6. Two prototypes and their superposition.

respect to modifications of the basic transformation costs. Computing time is linear in the number of states generated (see Figure 7). Utilizing heuristic information as described in Section 3 leads to the expansion of fewer states. As an example, when matching the input pattern e in Figure 5 against prototype e in Figure 4,43 expansions are needed utilizing heuristic information while 83 expansions are needed otherwise.

The authors want to point out that their approach to graph matching should not be regarded as being primarily devoted to the matching of silhouettes, for which other methods are known from the literature, e.g. Kashyap and Oommen (1982). Instead, this problem domain has been used as just one example for testing and gaining general experience. Other applications of graph

25

20

15

computing time (s)

/ J f J

~r

f J /

20 40 60 80 100 120 140 160 #of s ta tes

Fig. 7. Time versus number of states.

matching in the field of medical image sequence analysis are currently under investigation (Bunke, Sagerer and Niemann (1982)).

Acknowledgement

The authors want to express their gratitude to Prof. K.S. Fu of Purdue University, West Lafayette, IN for many fruitful discussions on the topic of this paper.

References

Bunke, H., G. Sagerer and FI. Niemann (1982). Model based analysis of scintigraphic image sequence of the human heart, Proc. o f the NA TO ASI on Image Sequence Processing and Dynamic Scene Analysis, Braunlage/Harz, F.R. Germany, June/July 1982.

Cheng, J.K. and T.S. Huang (1981). Image recognition by matching relational structures. Proc. IEEE Comp. Soc. Conf. on PRIP, 542-547.

Corneil, D.G. and C.C. Gotlieb (1970). An efficient algorithm for graph isomorphism. J. Assoc. Comput. Mach. 17 (1).

Fu, K.S. (1982). Syntactic Pattern Recognition and Applica- tions. Prentice Hall, Englewood Cliffs, NJ.

Fu, K.S. and B.L. Bhargava (1973). Tree systems for syntactic pattern recognition. IEEE Trans. Compur 22, 1087-1099.

Kashyap, R.L. and B.J. Oommen (1982). A geometrical approach to polygonal dissimilarity and shape matching, Proc. 6th ICPR, 472-479.

Nilsson, N.J. (1980). Principles o f Artificial Intelligence. Tioga, Palo Alto, CA.

Pavlidis, T. (1977). Structural Pattern Recognition. Springer, Berlin-Heidelberg-New York.

Sanfeliu, A. (1980). Distance between attributed relational

252


graphs for pattern recognition, TR-EE 80-39, Purdue University, West Lafayette, IN.

Shapiro, L.G. and R.M. Haralick (1981). Structural descrip- tions and inexact matching. IEEE Trans. Pattern Anal. Mach. Intell. 3(5), 504-529.

Tsai, W.H. and K.S. Fu (1979). Error-correcting isomorphisms of attributed relational graphs for pattern analysis, IEEE Trans. Systems Man Cybernet. 9(12).

Tsai, W.H. and F.Y. Chang (1982). General error-correcting matching of relational graphs for pattern analysis. To appear.

253

Documents

Inexact graph matching for structural pattern recognition