H euristic Pre-Clustering Relevance Feedback in Attention-Based Image Retrieval Wan-Ting Su, Wen-Sheng Chu and Jenn-Jier James Lien Speaker: Wen-Sheng

Heuristic Pre-Clustering Relevance Feedback

in Attention-Based Image Retrieval

Wan-Ting Su, Wen-Sheng Chu and Jenn-Jier James Lien

Speaker: Wen-Sheng Chu

Robotics Lab. CSIE NCKU

Robotics Lab, CSIE NCKU

System InterfaceSystem Interface

Result View

Heuristic Pre-Clustering View

User can revise the clustering results manually

User can change the positive group number on his/her own

Query Image

Positive Feedback

Negative Feedback


System Overview

Wavelet Transformation

Low-LowSubband

Attended View Extraction

Image Database

Ranking by Euclidean Distance

User Feedback

?

Query Image

END

No

Yes

Best Matches

PCA

VQUser

Re-clustering

Ranking by GBDA Learning

Offline Module : Attention-Based Image Retrieval

Online Module : Heuristic Pre-Clustering Relevance Feedback

Feature Extraction from Attended View

HeuristicPre-clustering


Wavelet and Attended View Extraction

• To reduce the computational cost• Contrast extraction is applied to the wavelet

coefficient in the LL-subband.

) ,( ,, qpdCq

jiji

contrast value of pixel p at image location (i, j)

Gaussian distance

neighborhood of pixel (i, j)

1

0

1

0,0

1

0

1

0,0

1

1

M

i

N

jji

M

N

j

M

iji

M

jCC

y

iCC

x

1

0

1

0,

M

i

N

jjiC

attention center

Got saliency

map!


System Overview


Low-LowSubband


Image Database


User Feedback

?

Query Image

END

No

Yes

Best Matches

PCA

VQUser

Re-clustering







Visual Features Extraction

• Table1. 32 low-level visual features

Features Dimension

Color mean, standard deviation and skew in HSV space

9

Standard deviation of the wavelet coefficients in 10 pyramid de-correlated

sub-bands

10

13 statistical elements extracted from the edge map such as max fill time, max fork

count, etc.

13


System Overview


Low-LowSubband


Image Database


User Feedback

?

Query Image

END

No

Yes

Best Matches

UserRe-clustering






Got features!

PCA

VQ


Pre-Clustering

• Principal Component Analysis (PCA)

+• Vector Quantization algorithm (VQ)


User Re-clustering

User Re-clustering

System Pre-clustering Result User Re-clustering Result


System Overview


Low-LowSubband


Image Database


User Feedback

?

Query Image

END

No

Yes

Best Matches

UserRe-clustering






PCA

VQ


Re-weighting Scheme

• Group-Based Discriminant Analysis (GBDA)• Multiple positive and multiple negative classes• Clustering each positive class• Scattering the negative example away from each

positive class

Single Flower

Bouquets of Flowers

Negative Samples

Positive Samples


GBDA

WSW

WSWW

WTPN

T

Wmaxarg

c

iNiPN SS

1

Sw : the sum of the within-class scatter matrix of the positive groups

SPN is the sum of between-class scatter matrices of positive-to-negative

iCx

Tiii )m)(xm(xS

mi : the mean of the ith positive class Ci

c

iiw SS

1c: the number of positive groups

Dy

TiiNi )m)(ym(yS

D : a set of negative examples


Experiment Result (1)

• COREL image database• QS2: 1000 images from 10 selected categories• Each of 10 categories contains 100 images and is

used as queries.

1. Sunset 2. Flower 3. Car 4. Ape 5. Mountain

6. Penguin 7. Tiger 8. Bird 9. Horse 10. Building

Table 1. Image Categories in Query Set 2

N

N returns in top retrieved imagesrelevant precision



20.00%

25.00%

30.00%

35.00%

40.00%

45.00%

50.00%

55.00%

60.00%

10 20 30 40 50 60 70 80 90 100

Scope

Pre

cisi

on

Attention-Based System Global



0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

1 2 3 4 5 6 7 8 9 10

Category ID

Pre

cisi

on

Attention-Based System Global


Experimental Results (4)

Precision = 5/10Precision = 7/20

Query Image

First-time retrieval results




First-time feedbackresults




Second-time feedback results


Conclusion

• The major work in this study is integrating attention-based image retrieval with the relevance feedback algorithm using multiple positive and negative groups.

• The system guides the user in clustering positive feedbacks by providing heuristic pre-clustering results. Then the user can revise the clusters manually.


Experiment Result - Video Demo

Documents

H euristic Pre-Clustering Relevance Feedback in Attention-Based Image Retrieval Wan-Ting Su, Wen-Sheng Chu and Jenn-Jier James Lien Speaker: Wen-Sheng