Image Retrieval Part I (Introduction)
2
Image Understanding Functions
• Image indexing
• similarity matching
• image retrieval (content-based method)
3
Images containing similar colors
4
Images containing similar content
5
Image Database
6
Image Database
7
Variety of Similarity
• Similar color distribution
• Similar texture pattern
• Similar shape/pattern
• Similar real content
Degree of difficulty
Histogram matching
Texture analysis
Image Segmentation,Pattern recognition
Life-time goal :-)
8
Image Indexing
9
10
Query by Example I
11
Query by Example 2
““Find all paintings ofFind all paintings ofthis shape”this shape”
12
Retrieval by Sketch
PictureFiderPictureFidersearch enginesearch engine
13
QBIC - Query By Image Content
First and best known commercial image retrieval system• IBM Almaden Research Center• Commercial product
Search by shape, color, texture, keyword
Queries can be based on: • example images, • user-constructed sketches and drawings, • selected color and texture patterns
User can: • Select colors and color distributions from a color wheel, • Select textures from a predetermined selection, • Adjust relative weights among shape features
14
Content-based image retrieval
• simple features are not semantic entities• as opposed to words in a text• not easy to extract semantic content• choice of indexing elements is unclear
• low-level features• color, texture, shapes
• high-level features• a person on bicycle, a windsurfer, vase
• abstract features• triumph, sadness, happiness
15
16
Content of Image
• Image = array of pixel intensities• b/w: 1 bit• gray: 8 bits• colour: 24 bits• code: index in colour lookup table
17
Image Retrieval by Intensity
• pair-wise comparison
• histogram methods
• central moments
18
Pair-wise comparison
• feature vector = raw array of pixel intensities
• needs model of pixel intensity distance dC
• compared pictured must be same size
• considers spatial arrangement
• not concise
19
Histogram methods
• quantise intensity space into N bins H(i) is pixel proportion with intensity of bin i
• distance of images I and Q (query example):
histogram intersection
• simple, concise method• global feature: locality is lost
• Exercise: why must the histograms be normalized?
20
Histogram Encoding
21
Center Image Histogram
22
Window Histogram Encoding
23
Window + center encoding
24
Intensity central moments
• instead of a histogram compute three statistical numbers per image (or window)
• mean (average gray level)• variance (squared standard deviation)• 3rd central moment (skewness of the grey level distribution)
25
Intensity central moments
• mean, variance and 3rd central moment:
26
Intensity central moments
• in order to have numbers which are comparable in size for each moment we use the corresponding root
• we get a 3-dim feature vector:
27
Moments + spatial coherence
• moments are a global feature
• spatial coherence can be added by• windows• centering• windows+centering
28
summary (retrieval by intensities)
• paiwise comparison
• histograms
• central moments
• balance in locality through• windows, centering, windows+centering
29
Retrieval by color
• RGB color model
• color histograms
• color moments
• clustering in color space
30
The RGB color model
• mixture of intensities of red, green and blue
• for active devices such as monitors
• some examples
31
color histograms
• simple methods: three 1-d histograms; one 3d histogram• create a 1d histogram for each color component each: eg, one
for R, one for G, one for B
• divide 3d color space into 3d bins for histograms:RR GG BB
32
Color moments
• feature vector:
9 numbers per image or window• 3 moments per color channel:
33
Clustering in color space
• cluster the pixels in 3d color space
• features: centroids and population of clusters
• similarity measure• compares color centriods and population• must take different cluster number into account
34
Summary (retrieval by color)
• RGB color model (or HSV color model)
• two color histograms
• moments of color
• clustering in color space
35
Texture
• perceptual phenomenon
• local region quality, not point quality
• depends on the scale
• texture = repeated pattern• local variations in image intensity• too fine as to be considered as own objects
• Texture feature• Co-occurence matrix• Gabor Wavlet Filter coeffiecients
36
Texture Examples
37
Segmentation of Images
• texture analysis needs segmentation of the still image to be meaningful
• separate objects• ignore background• assign texture feature per object
• object segmentation is a difficult task; edge detection can be useful for this
38
Texture+Color Segmentation
39
Shape Analysis
• Shape=
class of geometric objects invariant under
• information preserving description (for compression)
• non-information preserving (for retrieval)• boundary based (ignore interior)• region based (boundary+interior)
• Shape features• Chaincodes (boundary based)• Fourier Descriptor (boundary based)• ect
40
Examples of shapes
41
Chaincodes
• contour description by a string of directions
• Freeman chain code• f1f2f3...• eight possible directions for each pixel• translation invariant, not rotation invariant
42
Chain code histograms
• used for distance measure
• same histogram represents many shapes
• percentage histogram is scale invariant
43
Retrieval by shape
44
Retrieval by shape
45
Content-Based Image Retrieval
• many different low-level features can be computed, why did we do this?
• to describe a complex multimedia object• to automatically annotate mm objects with salient properties
(eg, color shape, texture)• to be able to compute similarities of mm objects• to make mm objects searchable by content• high-level feature are not (yet) accessible, but research
attempts to construct high-level concepts from low-level features
46
Content-Based Image Retrieval
• Which features are best for searching? Depends on the information need:
• looking for sunset holiday picture in your digital shoebox? Use color histograms
• want to build a wallpaper customer database? Use color+texture
• want to build a b/w sketch database for technical industrial designs? Use shape descriptors
• Not sure which features are best for a query (eg, if you also have abstract features such as Fourier coefficients)? Deploy relevance feedback and let the system learn the relevant feature for this query...
47
Content-based Image Retrieval
• Summary: we looked at• retrieval by intensity• color models• retrieval by color• retrieval by shape• retrieval by texture
• these features can be used in a general framework for content-based retrieval as discussed earlier
48
Experiment: Setting up CBIR search engine
• Create directory c:/Experiments/database
• copy database file “texturedatabase” into the directory
• copy the following two subdirectory into c:/Experiments/database
• ImageDatabase• ShapeColor
• Open database (“java COM.cloudscape.tools.cview”)
• install the database “texturedatabase”
• Close database
49
Compile file.Java & deploy application
• open cmd and type the following command• javac *.java
• run J2ee server
• run deploytool
• deploy your application to include the following Web Components:
• index.jsp• MyLocalesRbf2• MyDateJose2• and all the jpeg images stored in the directory called
“ImageDatabase”.
50
Summary of Fundamental CBIR
• Content-based image retrieval (CBIR)
• CBIR Algorithm:• First step --- Image Indexing
• Second step --- content matching