Advanced relevance feedback strategy for precise image retrieval

International Journal of Computer Engineering and Technology (IJCET), ISSN 0976-6367(Print),

ISSN 0976 - 6375(Online), Volume 6, Issue 1, January (2015), pp. 61-71© IAEME

61

ADVANCED RELEVANCE FEEDBACK STRATEGY FOR

PRECISE IMAGE RETRIEVAL

A. A. Khodaskara, S. A. Ladhake

b

aComputer Science and Engineering Department, SIPNA COET, Amravati, Maharashtra, India

bComputer Science and Engineering Department, SIPNA COET, Amravati, Maharashtra, India

ABSTRACT

Relevance feedback is effective technique for bridging the semantic gap in image retrieval

which diminish semantic gap between low-level visual features and high-level semantic concepts for

image retrieval. Currently, crucial image retrieval system is content-based image retrieval. To

improve performance of proposed content based image retrieval system, automatic

relevance feedback technique is proposed which based on inductive learning involve inductive

concept learning by decision tree. We implement and tested proposed image retrieval system which

refine the retrieval result as per user requirement until get required accuracy. Experimental result

shows improved performance in term of precision, recall and accuracy.

Keywords: Content based image retrieval, Decision tree, Inductive learning, Inductive concept

learning, Relevance feedback, Semantic gap.

1. INTRODUCTION

As we know, an immense growth and development in internet technology give birth to

gigantic digital images which are exploited in variety of applications such as medical, academic,

virtual museums, military etc. it is really difficult for users to search large image databases for

particular required image. In order to address this requirement, initially text based and then content

based image retrieval system is proposed. In text based image retrieval system, images are retrieved

on the basic of text annotation. Text based retrieval systems are suffered from low precision, recall

and accuracy. To overcome these limitations, Content based image retrieval system is introduced. In

content based image retrieval system, initially visual features such as color, texture and shape are

extracted from images and store in feature vector which are used to represent image. When user fired

an image query, then a query model converts the image into a feature vector of query image and

retrieval module performs image retrieval by computing similarities between query image and

images in image database. Finally, the retrieval results are ranked according to computed similarity

INTERNATIONAL JOURNAL OF COMPUTER ENGINEERING &

TECHNOLOGY (IJCET)

ISSN 0976 – 6367(Print)

ISSN 0976 – 6375(Online)

Volume 6, Issue 1, January (2015), pp. 61-71

© IAEME: www.iaeme.com/IJCET.asp

Journal Impact Factor (2015): 8.9958 (Calculated by GISI)

www.jifactor.com

IJCET

© I A E M E



62

values. Content based image retrieval systems are also critically suffer by semantic gap which

address by using relevance feedback mechanisms.

Relevance feedback mechanism provides way to interact human and computer to remodel

high-level queries to low-level features. Relevance feedback is a potent skill used to improve

accuracy and speed of retrieval in content based and text-based retrieval systems. Relevance

feedback is an iterative method, which filter and improve the retrieval result based on the previously

retrieved results user's feedback. Traditional there are two types of RF, short term RF and Long term

RF. Some retrieval system used merging of short-term and long-term learning with virtual feature

[8]. Relevance feedback automatically adjust an active query using the information provided by the

user about the earlier retrieved objects considering low-level features can capture high-level

concepts, the relevance feedback mechanism attempt to form the relation between high-level

concepts and low-level features from the user’s feedback.

1.1. Types of Relevance Feedback

Figure 1. Types of Relevance Feedback

1.2. Challenges Faced by Relevance Feedback

As mention earlier, relevance feedback is used to improve accuracy of image retrieval. There

are some challenges which need to address to improve relevance feedback strategy such as high level

feature extraction, imbalance feedback samples and real time processing.

1) High Level Semantics Extraction:

Image retrieval system is suffered from semantic gap. Extraction of high-level semantics from

images is difficult because RF used only low-level features.

2) Imbalance of feedback samples:

Machine learning algorithms are unable to accurate result, because smaller numbers of feedback

samples are labeled by the users during RF iteration which are far smaller feature space dimension.

Labelling of negative samples is generally greater than positive samples which result in imbalance of

training data. Thus the small positive sample definitely confines RF accuracy.

3) Real time processing:

Online relevance feedback takes more time which is addressed by implementing efficient image

representation and storage structures.



63

Figure 2. Challenges in content based image retrieval

2. ROLE OF RF IN CBIR

A number of relevance feedback techniques have been designed as a key tool to bridge the

semantic gap in content based image retrieval, and thus to improve the performance of CBIR

systems. By using relevance feedback scheme, a CBIR system learns from feedback given by the

user. Learning in CBIR systems is categorized into short-term learning, and long term learning [21].

• Short-term learning

CBIR system refines and tunes query using relevance feedback in single retrieval iteration. This type

of learning is usually known as intra-query learning or short term learning. A lot of standard machine

learning techniques are used in short-term learning, which include support vector machines,

Bayesian learning, boosting and feature weighting, discriminant analysis etc. short term learning is

online, which takes additional real time.

• Long term learning

The learning strategy analyzes the relationship between the current and past retrieval

iterations. This form of learning is inter-query or long term learning. Long-term learning methods

overcome the problem of short term learning. It confiscates inability of capturing semantics and

imbalance of feedback examples, and memory mechanism. It based on collaborative filtering.

3. ADVANCED DEVELOPMENT IN RF

This section highlights advance development in relevance feedback technique such as

Navigation-Pattern-based Relevance Feedback, Click-based relevance feedback, pseudo-relevance

feedback for multimedia retrieval, user historical feedback log, semi-supervised long-term

Relevance Feedback algorithm, a gaze-based Relevance Feedback, Gaussian based relevance

feedback, Advanced long-term relevance feedback system generalized biased discriminant analysis

relevance feedback algorithm etc.



64

View-based 3D model retrieval system implemented a relevance feedback method to balance

the contributions of multiple topic models with specified numbers of topics [1]. View-based 3D

model retrieval uses a set of views to represent each object. Click-based relevance feedback is

proposed which emphasizes the successful use of click-through data for identifying user search

intention [2]. Leverage click-through data can be considered as an implicit user feedback and help to

understand the query. Spoken term detection is a basic technology for spoken content retrieval,

which retrieve and browse multimedia content over the Internet. The concept of pseudo-relevance

feedback used in multimedia retrieval, which divide some spoken segments in the first-pass retrieved

results into relevant or pseudo-relevant and irrelevant or pseudo-irrelevant [3]. Adaptive wavelet-

based image characterizations used to characterize each query image which improves the retrieval

performance. Flexibility in wavelet adaptation also enhanced relevance feedback on image

characterization itself [4]. A novel subspace learning framework for learning an effective semantic

subspace is proposed on user historical feedback log data for a Collaborative Image Retrieval [5].

Basically, CPSL can efficiently assimilate discriminative information and geometrical information of

labeled log images, and the weakly similar information of unlabeled images together to learn a

reliable subspace.

Navigation-Pattern-based Relevance Feedback (NPRF) achieves the high efficiency and

effectiveness of CBIR in coping with the large-scale image data [6]. Proposed NPRF Search

discovered navigation patterns and three kinds of query refinement strategies, Query Point

Movement, Query Reweighting and Query Expansion, to match the user's goal successfully. A semi-

supervised long-term Relevance Feedback algorithm is designed which refine the multimedia data

representation and utilizes both the multimedia data distribution in multimedia feature space and the

history RF information provided by users [8]. Geometric optimum experimental design is a novel

active learning method selects multiple representative samples in the database as the most

informative ones for the user to label which improve relevance feedback [9]. A gaze-

based Relevance Feedback approach to region-based image retrieval is presented [12]. Proposed

object-based RF overcome the major drawback of region-based RF approaches, i.e., the frequently

inaccurate estimation of the regions of interest in the retrieved images. A biased maximum margin

analysis (BMMA) and a semisupervised BMMA (SemiBMMA) are proposed for integrating the

distinct properties of feedbacks and exploiting the information of unlabeled samples for SVM-based

RF schemes [14]. The BMMA distinguishes positive feedbacks from negative ones based on local

analysis, whereas the SemiBMMA can efficiently incorporate information of unlabeled samples by

introducing a Laplacian regularizer to the BMMA. Advanced long-term relevance feedback system

was proposed which integrates the user feedback from all iteration and instills memory into

the feedback system of content based retrieval without earlier retrieval log [16]. A new

relevance feedback approach for content-based image retrieval was introduced, which uses Gaussian

mixture (GM) models as image representations [17]. Proposed RF technique categorizes the relevant

and irrelevant images according to the preferences of the user. The generalized biased discriminant

analysis algorithm is the most promising relevance feedback way employed for content-

based image retrieval [18]. It evades the singular problem by adopting the differential scatter

discriminant criterion and manages the Gaussian distribution assumption.

New RF method with active sample-selecting and manifold learning was proposed for CBIR of SAR

images with limited feedback samples [19]. Particle swarm optimizer which merges

relevance feedback with an evolutionary stochastic algorithm was proposed which grasp user's

semantics through optimized iterative learning [20]. For improvement of performance, a real-time

textual query-based personal photo retrieval system for millions of Web images implemented

two relevance feedback methods using cross-domain learning, which efficiently employ both the

Web images and personal images [22].



65

4. PROPOSED SYSTEM

Proposed system has basic four components, image DB, features extraction, similarity

measure, and relevance feedback.

• Image database:

We proposed and tested image retrieval system based Inductive Learning strategy using

image database of 2400 images. Used image database consist of both image natural as well as

synthetic images.

• Feature extraction:

Feature extraction consists of extracting the significant information from the images. Once

the low level features are extracted, they are stored in the form of feature vector in Database. Feature

extraction is significantly reduces the information storage space as it represent an image for

understanding using image content. In the CBIR system, low level features of image such as color,

texture and shape are extracted using Discrete Wavelet Transform. Low level contents of the images

in the database are extracted and described by multi-dimensional feature vectors in image feature

database.

Figure 3. Workflow of proposed system

• Similarity measure:

In similarity measure, feature vector of user query image and database image feature vector

are compared using the Euclidean distance metric. The images are ranked based on the distance

value. For similarity measure, proposed system use Euclidean distance techniques for matching user



66

query image with images in database. Similarity measure is based on threshold vales of image

extracted features. If a Euclidean distance value is less than threshold value than image is irrelevant

otherwise image is relevant to query image.

• Advanced Relevance feedback strategy:

Basically, relevance feedback is a technique used in image retrieval to help the users for

retrieving the correct images from large database as per user requirement. The users can continue to

refine the search from the results of the previous search until they get the desired images or closest to

what they desire. In this framework, the Relevance Feedback functionality is described. The

Relevance Feedback is a composition of Inductive Learning Framework, Inductive Concept

Learning by Decision Tree.

Figure 4. CBIR with Relevance Feedback

a) Inductive Learning Framework:

Raw input data is processed to obtain a feature vector, x, that adequately all of the relevant

features for classifying examples. Each x is a list of (attribute, value) pairs. For example,

x = (Person = Sue, Eye-Color = Brown, Age = Young, Sex = Female)

The number of attributes (also called features) is fixed (positive, finite). Each attribute has a fixed,

finite number of possible values. Each example can be interpreted as a point in an n-dimensional

feature space, where n is the number of attributes.

b) Inductive Concept Learning by Learning Decision Trees

Build a decision tree for classifying examples as positive or negative instances of a concept.

Supervised learning, batch processing of training examples are employed using a preference bias. A

decision tree is a tree in which each non-leaf node has associated with it an attribute (feature), each

leaf node has associated with it a classification (+ or -), and each arc has associated with it one of the

possible values of the attribute at the node where the arc is directed from. For example,



67

Color

/ | \

/ | \

green/ red \blue

/ | \

Size + Shape

/ \ / \

/ \ / \

big/ small round \square

/ \ / \

- + Size -

/ \

/ \

big/ \small

/ \

- +

Figure 5. Learning Decision Tree

Algorithm:

Function decision-tree-learning (examples, attributes, default)

;; examples are a list of training examples

;; attributes is a list of candidate attributes for the

;; current node

;; default is the default value for a leaf node if there

;; are no examples left

If empty (examples) then return (default)

If same-classification (examples) then return (class (examples))

If empty (attributes) then return (majority-classification (examples))

best = choose-attribute(attributes, examples)

tree = new node with attribute best

for each value v of attribute best do

v-examples = subset of examples with attribute best = v

subtree = decision-tree-learning (v-examples, attributes - best, majority-classification(examples))

add a branch from tree to subtree with arc labeled v

return (tree)

5. RESULT EVALUATION

Proposed automatic relevance feedback strategy is presented and tested to reduce the gap

between low-level visual features and high-level semantic during the process of relevance feedback.

We implemented and tested the proposed content based image retrieval system with advanced

relevance feedback based on Inductive Learning Framework using a 2400 image subset from the

COREL image database. The images used are assorted set comprising both natural and man-made

scenes and objects. As proposed system is based on color, texture and shape feature abstraction,

equity in terms of these low level features was desired. Therefore, the test set used required striking a

balance of low level image content. The main objective of experimentation was to compare and

refine retrieval result to achieve high degree of retrieval accuracy.



68

In the first phase as shown in figure.7, proposed system retrieved a set of relevant image for user

query which has low accuracy as per user expectation. To improve accuracy of image retrieval,

proposed relevance feedback technique used to refine result as per user requirement as shown in

figure 8. During each iteration accuracy of retrieval is improves as shown in table1 which ultimately

reduce semantic gap. Performance of proposed image retrieval system evaluated using parameters

such as precision recall and accuracy with dataset of 2400 images. Figure 6 shows improvement of

precision with iteration.

Figure 6. Precision per iteration for user query

Table 1.Precision with relevance feedback Query Image Relevance Feedback

Iteration 1 Iteration 2 Iteration 3 Iteration 4

White rose 80.5 86.67 88.65 89.50

White Horse 77.45

89.91

90.46 91.56

Mountain 76.28 88.79 89.47 90.68

Beach 79.46 85.33 91.90 92.55

Figure 7. Result for user query



69

Figure 8. Result for user query with relevance feedback

6. CONCLUSION

We proposed automatic relevance feedback which reduces the semantic gap between low-

level features and high level concepts. We proposed content based Image retrieval system with

automatic relevance feedback which is an effective tool to improve the performance of CBIR in term

of accuracy, recall and precision. Proposed automatic relevance feedback is based on a composition

of Inductive Learning Framework, Inductive Concept Learning by Decision Tree. We used image

dataset of 2400 image to evaluate the result. Experimental result clearly shows improvement in

precision and accuracy of image retrieval.

REFERENCES

1. Leng, B. , Zeng, J. , Yao, M. , Xiong, Z., 3D Object Retrieval With Multitopic Model

CombiningRelevance Feedback and LDA Model, IEEE Trans. Image Processing, Vol.24

, No. 1, pp. 94 – 105, 2015.

2. Yongdong Zhang , Xiaopeng Yang , Tao Mei, Image Search Reranking With Query-

Dependent Click-Based Relevance Feedback, IEEE Trans. Image Processing, Vol.23 , No.

10, pp. 4448 – 4459, 2014.

3. Hung-Yi Lee , Lin-Shan Lee, Enhanced Spoken Term Detection Using Support Vector

Machines and Weighted Pseudo Examples, IEEE Trans. Audio, Speech, and Language

Processing, Vol.21 , No. 6, pp.1272 – 1284, 2013.



70

4. Quellec, G. , Lamard, M. , Cazuguel, G. , Cochener, B. , Roux, C., Fast Wavelet-Based

Image Characterization for Highly Adaptive Image Retrieval, IEEE Trans. Image Processing,

Vol. 21 , No. 4, pp. 1613 - 1623 , 2012.

5. Lining Zhang, Lipo Wang, Weisi Lin, Conjunctive Patches Subspace Learning With Side

Information for Collaborative Image Retrieval, IEEE Trans. Image Processing, Vol.21, No. 8,

pp. 3707 - 3720, 2012.

6. Ja-Hwung Su , Wei-Jyun Huang , Yu, P.S. , Tseng, V.S., Efficient Relevance Feedback for

Content-Based Image Retrieval by Mining User Navigation Patterns, IEEE Trans.

Knowledge and Data Engineering, Vol.23 , No. 3, 360 – 372, 2011.

7. Junwei Han, McKenna, S.J., Query-dependent metric learning for adaptive, content

based image browsing and retrieval, IEEE Trans. Image Processing, Vol.8, No. 10, pp. 610 –

618, 2014.

8. Peng-Yeng Yin, Bhanu, B. ; Kuang-Cheng Chang ; Anlei Dong, Long-Term Cross-Session

Relevance Feedback Using Virtual Features, IEEE Trans. Knowledge and Data Engineering,

Vol. 20 , no. 3, pp. 352 – 368, 2008

9. Lining Zhang, Lipo Wang, Weisi Lin, Shuicheng Yan , Geometric Optimum Experimental

Design for Collaborative Image Retrieval, IEEE Trans. Circuits and Systems for Video

Technology, Vol.24 , No. 2, pp.346 – 359, 2014.

10. Hao Ma , Jianke Zhu , Lyu, M.R.-T. , King, I., Bridging the Semantic Gap

Between Image Contents and Tags, IEEE Trans. Multimedia, Vol.12, No. 5, pp. 462 – 473,

2012.

11. Zilong Chen, Yang Lu, Using Text Classification Method in Relevance Feedback,

Springer LNCS Intelligent Information and Database Systems, Vol. 5991(2010) pp. 441-449

12. Papadopoulos, G.T. , Apostolakis, K.C. , Daras, P., Gaze-Based Relevance Feedback for

Realizing Region Based Image Retrieval, IEEE Trans. Multimedia, Vol.16 , No. 2, pp. 440 –

454, 2014.

13. Rahman, M.M., Antani, S.K., Thoma, G.R., A Learning-Based Similarity Fusion and

Filtering approach for Biomedical Image Retrieval Using SVM Classification

and Relevance Feedback, IEEE Trans. Information Technology in Biomedicine, Vol.15, No.

4, pp. 640 – 646, 2011.

14. Lining Zhang , Lipo Wang , Weisi Lin, Semisupervised Biased Maximum Margin Analysis

for Interactive Image Retrieval, IEEE Trans. Image Processing, Vol.21 , No. 4, pp. 2294 –

2308, 2012.

15. Imen Taktak, Mohamed Tmar, A. B. Hamadou, Query Reformulation Based on Relevance

Feedback, Springer LNCS Flexible Query Answering Systems, Vol. 5822, pp. 134-144,

2009.

16. Lakshmi, A., Nema, M. , Rakshit, S., Long Term Relevance Feedback: A Probabilistic Axis

Re-Weighting Update Scheme, IEEE Trans. SignalProcessingLetters,Vol.22, No.7, pp. 852 –

856, 2015.

17. Marakakis, A. , Siolas, G. , Galatsanos, N. , Likas, A. ,Stafylopatis,

A., Relevance feedback approach for image retrieval combining support vector machines and

adapted gaussian mixture models, IEEE Trans. Image Processing, Vol.5 , No. 6, pp. 531 –

540, 2011.

18. Lining Zhang, Lipo Wang, Weisi Lin, Generalized Biased Discriminant Analysis for

Content-Based Image Retrieval, IEEE Trans. Systems, Man, and Cybernetics, Part B:

Cybernetics, Vol.42 , No. 1, pp. 282 – 290, 2012.

19. Chen, R. , Cao, Y.F. , Sun, H., Active sample-selecting and manifold learning-based

relevance feedback method for synthetic aperture radar image retrieval, IEEE Trans. Radar,

Sonar & Navigation, Vol.5 , No. 2, pp. 118 – 127, 2011.



71

20. Broilo, M. , De Natale, F.G.B., A Stochastic Approach to Image Retrieval Using Relevance

Feedback and Particle Swarm Optimization, IEEE Trans. Multimedia, Vol.12, No. 4, pp. 267

– 277, 2010.

21. Gu, Hong, Zhao, Guangzhou, Qiu, Jun, Online metric learning for relevance feedback in e-

commerce image retrieval applications, IEEE Trans. Tsinghua Science and

Technology, Vol.16 , No. 4, pp. 377 – 385, 2011.

22. Yiming Liu , Dong Xu , Tsang, I.W. , Jiebo Luo, Textual Query of Personal Photos

Facilitated by Large-Scale Web Data, IEEE Trans. Pattern Analysis and Machine

Intelligence, Vol.33 , No. 5, pp. 1022 – 1036, 2011.

23. Tarun Dhar Diwan and Upasana Sinha, “Performance Analysis Is Basis on Color Based

Image Retrieval Technique” International journal of Computer Engineering & Technology

(IJCET), Volume 4, Issue 1, 2013, pp. 131 - 140, ISSN Print: 0976 – 6367, ISSN Online:

0976 – 6375.

24. Dr. Prashant Chatur, Pushpanjali Chouragade, “Visual Rerank: A Soft Computing Approach

For Image Retrieval From Large Scale Image Database” International journal of Computer

Engineering & Technology (IJCET), Volume 3, Issue 3, 2012, pp. 446 - 458, ISSN Print:

0976 – 6367, ISSN Online: 0976 – 6375.

25. Shivamurthy R.C, Manjunatha M B and Pradeep Kumar B.P, “A Comparative Study on

Procedures on Content-Based Image Retrieval In Medical Imaging” International journal of

Computer Engineering & Technology (IJCET), Volume 5, Issue 10, 2014, pp. 64 - 73, ISSN

Print: 0976 – 6367, ISSN Online: 0976 – 6375.