Dirk Nuemann and Karl R. Gegenfurtner 2002

Dirk Nuemann and Karl R. Gegenfurtner

2002

Perception Based Image Retrieval

Washington University in St. LouisMedia & Machines Lab

2Reynold Baileyhttp://www.cs.wustl.edu/~rjb1/

Introduction

Image indexing system based on known properties of the early stages of human vision.

Database of 60,000 photographs.

Find a set of images that is perceptually similar to some target image.



Color and Spatial Indexing

For each image the distribution of the features hue, luminance and spatial frequency were determined.

Summarized as three histograms (feature vectors)

Similarity of two images is the distance between the feature vectors.

Modeling of the appropriate image features was done in DKL color space.

Chromaticity and luminance stored as two separate indices.

Information about Fourier energy distribution was stored in a third histogram.



Color and Luminance Indexing

Color distribution of an image plotted w.r.t. the coordinates of the color opponent axes in DKL color space.



Conversion from Cartesian color opponent coordinates to polar coordinates:

ybirgsat

ybirghue

)arg( (Color angle)

(Distance from neutral gray)




Color histograms were created by dividing the chromaticity plane into 127 logarithmic-radial bins.




The size of the bins reflect the granularity of higher order perception:

For saturated colors, the resolution of hue is much finer than for unsaturated colors.

6 rings are used to discriminate saturation.

Color values exceeding 95% or falling below 5% of the maximally possible luminance value are classified as white and black respectively.

The ring a color belongs to is calculated using the logarithm of the saturation.

satr 2log

129 bins, 64 for the most saturated colors.





To calculate the color and luminance indices:

The color distribution of each image as mapped into these 129 bins.

For each image two vectors were stored.

• The frequency of the color tones f.• The average luminance level of the pixels in each bin l.

If bin is empty l is set to zero.




Frequency of the color bins for image with hats. The brightness of the bins is proportional to their frequency in the image. The most frequent is shown in white.




Color tones belonging to each bin are plotted with the average luminance of that bin (hats image).



Fourier Index

2D discrete Fourier transformation (DFT) was used to determine the distribution of Fourier energy across different orientations.

Done only in the luminance dimension of the DKL color space.

DFT determines the amplitude for a set of 2D sine waves. These basis functions can be characterized by a single complex parameter.

The polar coordinates of this parameter give the orientation and spatial frequency of the 2D sine waves.

The resulting Fourier spectrum was divided into radial logarithmic bins.



Fourier Index

Spatial frequency increases from the center to the edge of the chart. The brightness of a bin is proportional to the logarithm of the average energy within the bin. Highest energy is white. (for hats image).

126 bins

Each bin represents contrasts of distinct orientation and frequency ranges.



Distance metrics

Euclidean distance is used to compare the luminance and Fourier indices.

For chromaticity, the similarity between two images is given by the sum of the minimum of the corresponding bin frequencies.

n

i ii yxs1

),min(

Interpreted as the proportion the two color distributions share.

0.75 means that for 75% of the pixels in one image, there exists pixels in the other image which fall into corresponding color bins.



Distance metrics



Demo



Psychophysical Evaluation

Measure the relationship between the perceived similarity and the computed similarity.

Done for each of the three indices and for combinations of the three indices.

Two-alternative forced choice configuration.

Similarity (judged) of an image to the query image can be defined as the probability of selecting that image.



Psychophysical Evaluation

900 query images randomly selected from database.

For each of these the 2000 best matching images for each relevant index of index combination were retrieved.

The images with rank 2, 20, 200 and 2000 were selected for the experiment.

Self paced without decision time limits.



Experiment 1

For each of three distractor images, the rate of preferring the target is shown on the y axis.

Changing distractor similarity does not change the gradient but shifts the curves by constant amounts.



Experiment 2



Experiment 3

5% increase in judgments when spatial information included. Further 4% increase when luminance is included.



Questions

?

Documents

Dirk Nuemann and Karl R. Gegenfurtner 2002