1

A Compact Feature Representation and Image Indexing in Content-

Based Image Retrieval

A presentation by

Gita DasPhD Candidate

29 Nov 2005Supervisor: Dr. Sid Ray

Clayton School of Information TechnologyMonash University, Australia

Email: Gita.Das@csse.monash.edu.au

2

Overview

Introduction to CBIR

Research Issues

Feature Representation

Experimental Results

Conclusion and Future Directions

References

3

CBIR-what is it?

Each image is described by it’s visual features e.g. colour, shape, textureImage content is extracted e.g. Colour Histogram, Colour MomentsEach image is being represented by a M-dimensional feature vectorA similarity measure is used to find the distance between a query image and the database imageImages are ranked in order of closeness to query and top Nr images are returned to the user

4

CBIR-how does it work?

Image Database Feature Database

Query Image Feature Extraction

Feature Extraction

Similarity Measure Results

5

Research Issues

Improvement of System Accuracy– Proper selection of features and their representation– Use of multiple features & how to integrate them

Reduction of Semantic Gap– Human Intervention-Relevance Feedback– How to perceive user’s need, extract information and

incorporate user’s feedback into the system

Reduction in Retrieval time– Reduction in feature dimension– Efficient indexing

6

Feature Representation

Which colour model? We use HSV model which is perceptually uniform.Which colour representation? We use Colour Co-occurrence Matrices (CCM) of H, S, V space to construct a feature vector.What is a CCM? In a CCM,

P = [ pij], pij indicates the no. of times a pixel having colour level i co-occurs with another pixel having colour level j, at a position d. Why CCM? It not only gives pixel information but also spatial information of an image.

7

Sum-Average of CCM Elements

Haralick’s Sum-Average formula: If P is a LxL CCM,

);(.2

2

kpkSAL

kyx

p11 p12 p13 p14

p21 p22 p23 p24

p31 p32 p33 p34

p41 p42 p43 p44

SA=2p11+3(p21+p12)+4(p31+p22+p13) +……+8(p44)

L

i

L

jijyx pkp

1 1

)(

Where i+j=k, k=2,3,….2Land

……(1)

8

A compact feature vector

1

1 1

)(_L

i

L

ijijpjindiagSum

As we considered pixel pairs in both horizontal and vertical directions, H,S,V CCMs are symmetric.For H=16, S=3, V=3,Original dimension: 148-D (16+120+3+3+3+3)Reduced dimension: 25-D (16+1+3+1+3+1)

We used all diagonal elements of CCM. And a single Sum-average value to represent all non-diagonal elements as per following formula:

…….(2) where i,j are row and column no.

9

Image Indexing

Each image is represented by a 25-D feature vector.

Feature values are normalized to lie in the [0,1] range so that each component contributes equally in the distance metric.

We start with equal weights to all components and then update them using Relevance Feedback.

10

Similarity Measure

We used a weighted Minkowski distance to measure similarity between query image, Q and database image I:

||*),(1

iQiI

M

ii ffwQID

vector.feature ofdimension is andcomponent feature

for weight is wcomponent, feature is f where,

M

thii

thii

…….(3)

11

Relevance Feedback

kirel

kiNrk

iw,

,1

imagesrelevant over SD: images, retrieved over SD :

iteration in comp. feature of weight : where

kirel,

kiNr,

1

Nr

kiw ththki

.….(5)

RF is essential to reduce semantic gap.We updated both query vector and the weights in eqn. (3) as follows:

RN

lRili NRQ

1, / …….(4)

imagesrelevant of no. theis andlyrespective imagerelevant and imagequery

for component feature are and where

R

th

thl,ii

NliRQ

12

Experimental Results

No. of images in database: 2000No. of categories: 10 (Flowers, Fruits and Vegetables, Nature, Leaves, Ships, Faces, Fishes, Cars, Animals, Aeroplanes)Query Image: all 2000 images chosen as query and then averaged to get final precision.Precision is used to measure performance.

retrieved images of no.

retrieved imagesrelevant of no. Precision

13

Experimental Results

Fig. 1 Effect of H-only and H,S,V together on precision at different feature dimensions

14

Experimental Results

Fig 2: Precision is marginally worse at low scope and significantly better at higher scope

15

Experimental Results

Scope 148-D 25-D

20 16.09 16.272

200 9.12 12.583

Fig. 3 graphs showing improvement in precision with RF at different scopes andat different dimensionsTable shows increase of precision(%) from 0rf to 5rf.

16

Conclusion and Future Directions

What do we conclude?

Addition of S and V-space with H-space improves information content of images and hence precision

Less online computation time with our feature vector

Better precision with dimension reduction

Future work ?

Compare our method with other existing ones

Precision as a function of sample size and scope

RF as a multiple class problem as opposed to binary

17

References

1. Young Rui, Thomas S. Huang, Shih-Fu Chang, Image Retrieval: Current Techniques, Promising Directions and Open Issues, Journal of Visual Communication and Image Presentation, Vol. 10, No. 4, April 1999.

2. R. M. Haralick, K. Shanmugam, and I. Dinstein, “Textural features for image classification,” IEEE Transactions on Systems, Man, and Cybernetics, pp. 610–621, November 1973.

3. S. Aksoy and R. M. Haralick, F.A. Cheikh and M. Gabbouj, “A weighted distance approach to relevance feedback,” in International Conference on Pattern Recognition, Barcelona, Spain, September 2000.

4. S.-O. Shim and T.-S.Choi, “Image Indexing by modified colour co-occurrence matrix,” in International Conference on Image Processing, Vol. 3, September 2003.

18

THANK YOU!

Any Questions?