Image Retrieval Part I (Introduction). 2 Image Understanding Functions Image indexing similarity...

Image Retrieval Part I (Introduction)

2

Image Understanding Functions

• Image indexing

• similarity matching

• image retrieval (content-based method)

3

Images containing similar colors

4

Images containing similar content

5

Image Database

6

Image Database

7

Variety of Similarity

• Similar color distribution

• Similar texture pattern

• Similar shape/pattern

• Similar real content

Degree of difficulty

Histogram matching

Texture analysis

Image Segmentation,Pattern recognition

Life-time goal :-)

8

Image Indexing

9

10

Query by Example I

11

Query by Example 2

““Find all paintings ofFind all paintings ofthis shape”this shape”

12

Retrieval by Sketch

PictureFiderPictureFidersearch enginesearch engine

13

QBIC - Query By Image Content

First and best known commercial image retrieval system• IBM Almaden Research Center• Commercial product

Search by shape, color, texture, keyword

Queries can be based on: • example images, • user-constructed sketches and drawings, • selected color and texture patterns

User can: • Select colors and color distributions from a color wheel, • Select textures from a predetermined selection, • Adjust relative weights among shape features

14

Content-based image retrieval

• simple features are not semantic entities• as opposed to words in a text• not easy to extract semantic content• choice of indexing elements is unclear

• low-level features• color, texture, shapes

• high-level features• a person on bicycle, a windsurfer, vase

• abstract features• triumph, sadness, happiness

15

16

Content of Image

• Image = array of pixel intensities• b/w: 1 bit• gray: 8 bits• colour: 24 bits• code: index in colour lookup table

17

Image Retrieval by Intensity

• pair-wise comparison

• histogram methods

• central moments

18

Pair-wise comparison

• feature vector = raw array of pixel intensities

• needs model of pixel intensity distance dC

• compared pictured must be same size

• considers spatial arrangement

• not concise

19

Histogram methods

• quantise intensity space into N bins H(i) is pixel proportion with intensity of bin i

• distance of images I and Q (query example):

histogram intersection

• simple, concise method• global feature: locality is lost

• Exercise: why must the histograms be normalized?

20

Histogram Encoding

21

Center Image Histogram

22

Window Histogram Encoding

23

Window + center encoding

24

Intensity central moments

• instead of a histogram compute three statistical numbers per image (or window)

• mean (average gray level)• variance (squared standard deviation)• 3rd central moment (skewness of the grey level distribution)

25

Intensity central moments

• mean, variance and 3rd central moment:

26

Intensity central moments

• in order to have numbers which are comparable in size for each moment we use the corresponding root

• we get a 3-dim feature vector:

27

Moments + spatial coherence

• moments are a global feature

• spatial coherence can be added by• windows• centering• windows+centering

28

summary (retrieval by intensities)

• paiwise comparison

• histograms

• central moments

• balance in locality through• windows, centering, windows+centering

29

Retrieval by color

• RGB color model

• color histograms

• color moments

• clustering in color space

30

The RGB color model

• mixture of intensities of red, green and blue

• for active devices such as monitors

• some examples

31

color histograms

• simple methods: three 1-d histograms; one 3d histogram• create a 1d histogram for each color component each: eg, one

for R, one for G, one for B

• divide 3d color space into 3d bins for histograms:RR GG BB

32

Color moments

• feature vector:

9 numbers per image or window• 3 moments per color channel:

33

Clustering in color space

• cluster the pixels in 3d color space

• features: centroids and population of clusters

• similarity measure• compares color centriods and population• must take different cluster number into account

34

Summary (retrieval by color)

• RGB color model (or HSV color model)

• two color histograms

• moments of color

• clustering in color space

35

Texture

• perceptual phenomenon

• local region quality, not point quality

• depends on the scale

• texture = repeated pattern• local variations in image intensity• too fine as to be considered as own objects

• Texture feature• Co-occurence matrix• Gabor Wavlet Filter coeffiecients

36

Texture Examples

37

Segmentation of Images

• texture analysis needs segmentation of the still image to be meaningful

• separate objects• ignore background• assign texture feature per object

• object segmentation is a difficult task; edge detection can be useful for this

38

Texture+Color Segmentation

39

Shape Analysis

• Shape=

class of geometric objects invariant under

• information preserving description (for compression)

• non-information preserving (for retrieval)• boundary based (ignore interior)• region based (boundary+interior)

• Shape features• Chaincodes (boundary based)• Fourier Descriptor (boundary based)• ect

40

Examples of shapes

41

Chaincodes

• contour description by a string of directions

• Freeman chain code• f1f2f3...• eight possible directions for each pixel• translation invariant, not rotation invariant

42

Chain code histograms

• used for distance measure

• same histogram represents many shapes

• percentage histogram is scale invariant

43

Retrieval by shape

44

Retrieval by shape

45

Content-Based Image Retrieval

• many different low-level features can be computed, why did we do this?

• to describe a complex multimedia object• to automatically annotate mm objects with salient properties

(eg, color shape, texture)• to be able to compute similarities of mm objects• to make mm objects searchable by content• high-level feature are not (yet) accessible, but research

attempts to construct high-level concepts from low-level features

46

Content-Based Image Retrieval

• Which features are best for searching? Depends on the information need:

• looking for sunset holiday picture in your digital shoebox? Use color histograms

• want to build a wallpaper customer database? Use color+texture

• want to build a b/w sketch database for technical industrial designs? Use shape descriptors

• Not sure which features are best for a query (eg, if you also have abstract features such as Fourier coefficients)? Deploy relevance feedback and let the system learn the relevant feature for this query...

47

Content-based Image Retrieval

• Summary: we looked at• retrieval by intensity• color models• retrieval by color• retrieval by shape• retrieval by texture

• these features can be used in a general framework for content-based retrieval as discussed earlier

48

Experiment: Setting up CBIR search engine

• Create directory c:/Experiments/database

• copy database file “texturedatabase” into the directory

• copy the following two subdirectory into c:/Experiments/database

• ImageDatabase• ShapeColor

• Open database (“java COM.cloudscape.tools.cview”)

• install the database “texturedatabase”

• Close database

49

Compile file.Java & deploy application

• open cmd and type the following command• javac *.java

• run J2ee server

• run deploytool

• deploy your application to include the following Web Components:

• index.jsp• MyLocalesRbf2• MyDateJose2• and all the jpeg images stored in the directory called

“ImageDatabase”.

50

Summary of Fundamental CBIR

• Content-based image retrieval (CBIR)

• CBIR Algorithm:• First step --- Image Indexing

• Second step --- content matching