Information Processing Letters 75 (2000) 113–117

Computational complexity of similarity retrievalin a pictorial database

D.J. Guana,∗, Chun-Yen Choub, Chiou-Wei Chenba Department of Computer Science, National Sun Yat-Sen University, Kaohsiung 80424, Taiwan

b Department of Applied Mathematics, National Sun Yat-Sen University, Kaohsiung 80424, Taiwan

Received 12 January 2000; received in revised form 5 April 2000Communicated by K. Iwama

Abstract

Three types of similarity are commonly used in similarity retrieval of a pictorial database. Only one of them is isometric,hence transitive. We present polynomial time algorithms to find the maximum similar subpicture of two given pictures whenthe similarity is isometric. For nonisometric types, we show that finding a maximum similar subpicture in two given pictures isNP-complete, even when all symbols are distinct, by reducing the satisfiability problem to it. 2000 Elsevier Science B.V. Allrights reserved.

Keywords:Information retrieval; Pictorial database; Similarity retrieval

1. Introduction

Storing and retrieving of pictures in a databasecan be quite different than in conventional databasesthat handle only texts. To be able to retrieve picturesby their contents, descriptions of the pictures areoften stored together with the pictures. Tanimotointroduced iconic indexing to describe the contents ofa picture [5]. In short, symbols and their coordinatesare used to describe a picture. This allows the use ofa spatial relationship of symbols for the retrieval ofpictures. For example, we may specify a picture whichhas a tree in front of a house.

The picture itself and the symbolic descriptionof the picture are two different entities stored in a

∗ Corresponding author.E-mail addresses:guan@cse.nsysu.edu.tw (D.J. Guan),

chou@math.nsysu.edu.tw (C.-Y. Chou).

pictorial database. In this paper, a picture is defined asthe symbolic description of the picture, not the pictureitself.

Spatial relationship is a fuzzy concept depending onhuman interpretation. Thus, the specifications of pic-tures requested by a user may not be exact. Similar-ity retrieval of pictures, which is one of the distin-guishing functions different for a conventional data-base, must be provided in a pictorial database. Theremay be many criteria to measure the “similarity” be-tween two given pictures. In this paper, we considerthe case of finding subpictures of maximum size whichare “similar”. We will define this criterion formally inthe following paragraphs.

In general, there are eight directions which arecommonly used in two-dimensional pictures, namelyleft, right, up, down, upper left, upper right, lower left,and lower right. According to these eight directions,given two picturesP1 andP2, Chang et al. defined

0020-0190/00/$ – see front matter 2000 Elsevier Science B.V. All rights reserved.PII: S0020-0190(00)00092-2

114 D.J. Guan et al. / Information Processing Letters 75 (2000) 113–117

three types of similarity relation for any pair of objectsin P1 and P2 [1]. The three types of similarity areequivalent to the following definitions by using thecoordinates of the symbols.

Let A be the symbol assigned to an icon whosecoordinates are(x, y) in a pictureP . We call thesymbolA with coordinates(x, y) anobjectof P , andit is denoted by[A, (x, y)].

Definition 1. Let [A, (x1, y1)] and [B, (x2, y2)] betwo objects inP1, and[A, (a1, b1)] and [B, (a2, b2)]two objects inP2. Let 1x = x2 − x1,1y = y2 −y1,1a = a2− a1, and1b = b2− b1. We say that ob-jects[A, (x1, y1)] and[B, (x2, y2)] in P1 and objects[A, (a1, b1)] and[B, (a2, b2)] in P2 have a relation oftype-i, i = 0,1,2, according to the corresponding cri-terion.

type-0: 1x1a > 0 and1y1b > 0.

type-1: “1x1a > 0 or1x =1a = 0” and “1y1b >0 or1y =1b = 0”.

type-2: 1x =1a and1y =1b.

Definition 2. Let S be a subset of objects in pictureP1 andT be a subset of objects inP2. We say thatS andT are similar subpictures of type-i, i = 0,1,2,respectively, if there is a bijective functionf from S

to T such that:(1) For every object[A, (x, y)] in S, the first com-

ponent of[A, (x, y)] and the first component off ([A, (x, y)]) are the same. That is, if we considera picture to be a set of icons [5], they represent thesame type of icons.

(2) For any pair of objects[A, (x1, y1)] and [B, (x2,

y2)] in S, [A, (x1, y1)] and [B, (x2, y2)] in P1and f ([A, (x1, y1)]) and f ([B, (x2, y2)]) in P2have the spatial relation of type-i, i = 0,1,2,respectively.

The problem of finding maximum similar subpic-tures inP1 andP2 is to find a subset of objectsS inP1 and a subset of objectsT in P2 such thatS andT are similar subpictures of type-i, i = 0,1,2, withmaximum cardinality.

Intuitively, in finding maximum similar subpictures,the case when the symbols are all distinct is easierthan the case when the symbols are not all distinct.

Given two picturesP1 and P2. When symbols areall distinct in each picture, Chang et al. reduce theproblem of finding maximum similar subpictures oftype-i, i = 0,1,2, to the clique problem of a graphconstructed fromP1 and P2. A graphGi(P1,P2),i = 0,1,2, is constructed as follows. Let the vertexsetV be the set of symbols appearing in bothP1 andP2. There is an edge betweenA andB if and onlyif there are two objects[A, (x1, y1)], [B, (x2, y2)] inP1, and two objects[A, (a1, b1)], [B, (a2, b2)] in P2,such that they have the spatial relation of type-i, i =0,1,2, respectively. Therefore we have an undirectedgraphGi(P1,P2), and a maximum complete subgraphof Gi(P1,P2) corresponds to a maximum similarsubpicture of type-i, i = 0,1,2.

Note that similarity of type-2 is isometric, that is,the difference in both coordinates are preserved, hencetransitive. For maximum similar subpictures of type-2,we present an O(n2) algorithm when symbols are alldistinct and an O(n3) algorithm when symbols are notall distinct.

A related problem is called thepicture patternmatching problem. Given a pictureP and a patternQ,determine whether the patternQ appears in the pictureP or not. Tucci et al. [6] showed that type-0 and type-1picture matching problems are NP-complete when thesymbols are not all distinct.

Note that when all the symbols are distinct, wecan construct the graphGi(P,Q) and check if theinduced subgraphGi(P,Q)[U ] is a complete graph ornot, whereU is the subset of vertices correspondingto the patternQ. This shows that the picture patternmatching problem can be solved in polynomial timewhen all the symbols are distinct.

We show in this paper that picture similarity prob-lems of type-0 and type-1 are NP-complete even whenthe symbols are all distinct. Our method is to reducethe satisfiability problem to them.

2. Maximum similar subpictures of type-2

Assume that we are given two picturesP1 andP2each containing a set ofn distinct objects. Recallthat, to find a maximum similar subpicture of type-2,we can construct the corresponding undirected graphG2(P1,P2). We show thatG2(P1,P2) is a union ofcomplete subgraphs. Therefore, there is an efficient

D.J. Guan et al. / Information Processing Letters 75 (2000) 113–117 115

algorithm for finding maximum similar subpictures oftype-2.

Note that similarity of type-2 is transitive. Let[A, (x1, y1)], [B, (x2, y2)], and [C, (x3, y3)] be threedistinct objects inP1, and[A, (a1, b1)], [B, (a2, b2)],and[C, (a3, b3)] be three distinct objects inP2. Sinceall symbols are distinct, we may use the short notation“A and B have a relation of type-2” to mean that“ [A, (x1, y1)] and[B, (x2, y2)] in P1 and[A, (a1, b1)]and [B, (a2, b2)] in P2 have a relation of type-2”.Assume that(1) A andB have a relation of type-2, and(2) B andC have a relation of type-2.We can conclude thatA andC must have a relation oftype-2. In the graphG2(P1,P2), if there is an edgebetweenA and B and an edge betweenB and C,then there must be an edge betweenA andC. There-fore, the graphG2(P1,P2) is a union of complete sub-graphs. Hence, by checking the adjacency lists, it onlytakes O(n) time to identify each maximal completesubgraph. Since it only takes O(n2) time to constructthe graphG2(P1,P2), we have the following theo-rem.

Theorem 1. The problem of finding maximum similarsubpictures of type-2can be solved inO(n2) time whenall symbols are distinct.

When symbols are not all distinct, we have thefollowing algorithm, taking O(n3) time, to solve theproblem of maximum similar subpictures of type-2.

The algorithm first lists all objects inP1 and all ob-jects inP2, respectively, in the order of the sum of theirhorizontal and vertical coordinates and, when there isa tie, in the order of horizontal coordinates. The al-gorithm then finds a maximum subset of objectsS inP1 andT in P2, such thatS andT are similar sub-pictures. Suppose that[A, (x1, y1)] in S is mapped to[A, (a1, b1)] in T underf , that is,f ([A, (x1, y1)])=[A, (a1, b1)]. Then any object[B, (x2, y2)] in P1 and[B, (a2, b2)] in P2 with x2 − x1 = a2 − a1 andy2 −y2 = b2 − b1 must also be inS andT , respectively,andf ([B, (x2, y2)])= [B, (a2, b2)]. We shall call thetwo objects[B, (a2, b2)] in P1 and[B, (x2, y2)] in P2amatchwith respect to the pair of objects[A, (x1, y1)]in S and[A, (a1, b1)] in T .

The algorithm computesS andT by superimposi-tion. For each[si, (xi, yi)] in P1, letQsi be the subset

of objects inP2 whose first component issi . Then,for each[si, (aj , bj )] in Qsi , the algorithm counts thenumber of matches by superimposingP1 andP2. Thesuperimposing is done by identifying the coordinates(xi, yi) of P1 and (aj , bj ) of P2. That is, subtract1x = aj − xi from all horizontal coordinates of ob-jects inP2, and subtract1y = bj − yi from all verti-cal coordinates of objects inP2. Then the algorithmcounts the number of matches in the two modifiedlists. Since the original objects in each picture are or-dered, the counting of the number of matches can bedone in linear time by scanning the two lists in a waysimilar to merge sort.

It is easy to see that, for each object[si, (xi, yi)] inP1 and each[si, (aj , bj )] in Qsi , it takes O(n) time tocount the matches. Since the number of objects inP1

is at mostn, the number of objects inQsi is at mostn for eachsi , the counting of matches is performedat mostn2 times. Therefore the above algorithm is ofO(n3) time.

Theorem 2. The problem of finding maximum similarsubpictures of type-2can be solved inO(n3) time whensymbols are not all distinct.

3. Maximum similar subpicture problem of type-0or type-1

In this section, we show that finding the maximumsimilar subpicture of type-0 or type-1 is NP-complete.We reduce the satisfiability problem to them.

Let the set of Boolean variables be{x1, x2, . . . , xk}.Given a Boolean formulaΦ in conjunctive normalform, say,Φ = c1 ∧ c2 ∧ · · · ∧ cn, where for eachclausecr , r = 1,2, . . . , n, cr = (lr1 ∨ lr2 ∨ lr3) for somedistinct literalslr1, l

r2, l

r3. We construct two picturesP1

andP2 fromΦ as follows.P1 and P2 are both of size 3n in width and 2k

in height. Each picture will contain 3n objects, andthey are labeled aslr1, l

r2, l

r3, r = 1,2, . . . , n. The

coordinates of each object with symbollri in eachpicture are assigned as follows.• In pictureP1: [lri , (g1(i, r), h1(i, r))].• In pictureP2: [lri , (g2(i, r), h2(i, r))].

116 D.J. Guan et al. / Information Processing Letters 75 (2000) 113–117

The functionsg1, g2, h1, andh2 are defined as follows.

g1(i, r)=

3r − 2 if i = 1,

3r − 1 if i = 2,

3r if i = 3,

g2(i, r)=

3r if i = 1,

3r − 1 if i = 2,

3r − 2 if i = 3,

h1(i, r)= 2j if lri is the Boolean variablexj ,

2j − 1 if lri is the Boolean variablexj ,

h2(i, r)= 2j − 1 if lri is the Boolean variablexj ,

2j if lri is the Boolean variablexj .

The symbols arelri ’s, hence all distinct. Note that ittakes polynomial time to constructP1 andP2.

Example. If Φ = (x1∨ x3∨ x2)∧ (x1∨ x2∨ x3) withthe specified order of clauses and literals, then thecorrespondingP1 is

l12

l23

l13

l22

l21

l11

The correspondingP2 is

l23

l12

l22

l13

l11

l21

BetweenP1 andP2, the only differences are(1) within each clause, the order of columns for the

three literals are opposite;

(2) for two complementary literals, the order of rowsare opposite.

Therefore we have the following properties.

Property I. For the samer and i 6= j , the two ob-jects [lri , (g1(i, r), h1(i, r))], [lrj , (g1(j, r), h1(j, r))]in P1 and the two objects[lri , (g2(i, r), h2(i, r))],[lrj , (g2(j, r), h2(j, r))] in P2 don’t have a relation-ship of type-0 or type-1.

Property II. For r 6= s and anyi, j , if the Booleanvariables for lri and lsj are complementary, thenthe two objects[lri , (g1(i, r), h1(i, r))], [lsj , (g1(j, s),

h1(j, s))] in P1 and the two objects[lri , (g2(i, r),

h2(i, r))], [lsj , (g2(j, s), h2(j, s))] in P2 don’t have arelationship of type-0 or type-1.

Property III. For r 6= s and anyi, j , if the Booleanvariables for lri and lsj are not complementary, thenthe two objects[lri , (g1(i, r), h1(i, r))], [lsj , (g1(j, s),

h1(j, s))] in P1 and the two objects[lri , (g2(i, r),

h2(i, r))], [lsj , (g2(j, s), h2(j, s))] in P2 have a rela-tionship of type-0 or type-1.

Clearly, by Property I, any subset of objects with arelation of type-0 or type-1 betweenP1 andP2 hasat most one literal from each clause. Therefore itssize is at mostn. For brevity, we use maxi{P1,P2},i = 0,1, to denote the maximum size of subsets havinga relation of type-i betweenP1 and P2. Clearly,maxi{P1,P2}6 n, i = 0,1.

Lemma 1. Given two picturesP1 andP2.(1) Φ is satisfiable if and only ifmax0{P1,P2} = n.(2) Φ is satisfiable if and only ifmax1{P1,P2} = n.

Proof. If Φ is satisfiable, then there is a satisfy-ing assignment and each clausecr contains at leastone literal lri assigned true. Pick one such literalfrom each clause, and add the corresponding object[lri , (g1(i, r), h1(i, r))] in P1 to S and the correspond-ing object[lri , (g2(i, r), h2(i, r))] in P2 to T , we get asetS andT of sizen. Note that for any two objects[lri , (g1(i, r), h1(i, r))] and [lsj , (g1(j, s), h1(j, s))] inS, the Boolean variables forlri and lsj are not com-plementary. Therefore the two objects[lri , (g1(i, r),

h1(i, r))], [lsj , (g1(j, s), h1(j, s))] in S and the two ob-jects [lri , (g2(i, r), h2(i, r))], [lsj , (g2(j, s), h2(j, s))]

D.J. Guan et al. / Information Processing Letters 75 (2000) 113–117 117

in T have a relationship of type-0 by Property III.Hence S and T are similar subpictures of type-0.Therefore max0{P1,P2} = n.

If there is a subsetS of objects inP1 with sizen, and a subset of objectsT in P2 such thatS andT are similar subpictures of type-0, then for eachclausecr , S contains an object[lri , (ig1(i, r), h1(i, r))]where lri is a literal in the clausecr by Property Iand the pigeonhole principle. For any two objects[lri , (g1(i, r), h1(i, r))] and [lsj , (g1(j, s), h1(j, s))] inS, the Boolean variables forlri andlsj are not comple-mentary by Property II. Therefore, by assigning trueto each corresponding literal inS,Φ is satisfiable.

Similarly, we can show thatΦ is satisfiable if andonly if max1{P1,P2} = n. 2

Therefore, we have the following two lemmas.

Lemma 2.(1) The problem of finding maximum similar subpic-

tures of type-0 is NP-hard.(2) The problem of finding maximum similar subpic-

tures of type-1 is NP-hard.

Recall that, when symbols are all distinct, we canreduce the problem of maximum similar subpicturesof type-0 (respectively, type-1) to the maximum clique

problem. Therefore, both problems are in NP. We thenhave the following conclusions.

Theorem 3.(1) The problem of finding maximum similar subpic-

tures of type-0 is NP-complete even when all sym-bols are distinct.

(2) The problem of finding maximum similar subpic-tures of type-1 is NP-complete even when all sym-bols are distinct.

References

[1] S.K. Chang, Q.Y. Shi, C.W. Yan, Iconic indexing by 2D strings,IEEE Trans. Pattern Anal. Machine Intelligence 9 (3) (1987)413–428.

[2] S.K. Chang, C.W. Yan, D.C. Dimitroff, T. Arndt, An intelligentimage database system, IEEE Trans. Software Engrg. 14 (5)(1988) 681–688.

[3] G. Costagliola, F. Ferrucci, G. Tortora, M. Tucci, Cor-respondences—Non-redundant 2D strings, IEEE Trans. Knowl-edge Data Engrg. 7 (2) (1995) 347–350.

[4] S.Y. Lee, M.K. Shan, W.P. Yang, Similarity retrieval of iconicimage database, Pattern Recognition 22 (1988) 675–682.

[5] S.L. Tanimoto, An iconic symbolic data structuring scheme, in:Pattern Recognition and Artificial Intelligence, Academic Press,New York, 1976, pp. 452–471.

[6] M. Tucci, G. Costagliola, S.K. Chang, A remark on NP-com-pleteness of picture matching, Inform. Process. Lett. 39 (1991)241–243.