57
Indexing Techniques for Multimedia Databases Multimedia Similarity Search Structure Image Indexing Video Indexing

Indexing Techniques for Multimedia Databases Multimedia Similarity Search Structure Image Indexing Video Indexing

Embed Size (px)

Citation preview

Page 1: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

Indexing Techniques for Multimedia Databases

Multimedia Similarity

Search Structure

Image Indexing

Video Indexing

Page 2: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

2

Traditional DBMS

– Designed to manage one-dimensional datasets consisting of simple data types, such as strings and numbers

– Limited kinds of queries: exact match, partial match, and range queries

– Well-understood indexing methods: B-trees, hashing

Page 3: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

3

Characteristic of Multimedia Queries

• We normally retrieve a few records from a traditional DBMS through the specification of exact queries based on the notions of “equality”.

• The types of queries expected in an image/video DBMS are relatively vague or fuzzy, and are based on the notion of “similarity”.

The indexing structure should be able to satisfy similarity-based queries for a wide range of similarity measures.

Page 4: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

4

Content-Based Retrieval

• It is necessary to extract the features which are characteristics of the image and index the image on these features.

Examples: Shape descriptions, texture

properties.

• Typically there are a few different quantitative measures which describes the various aspect of each feature.

Example: The texture attribute of an image can be modeled as a 3-dimensional

vector with measures of directionality, contrast, and coarseness.

Page 5: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

5

Introduction

• Multimedia require support of multi-dimensional datasets– E.g., a 256 dimensional feature

vector.

• That implies– Specialized kinds of queries– New indexing approaches. Two

choices:• Map n-dimensional data to a single

dimension and use traditional indexing structures (B-trees)

• Develop specialized indexing structures

Page 6: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

6

Low-Dimensional Indexing Applications

• Spatial Databases (GIS, CAD/CAM)– Number of dimensions: 2-4– Spatial queries. For example:

• Which objects intersect a given 2D or 3D rectangle

• Which objects intersect a given object

– Specialized indexing structures• quad-tree, BSP-tree, K-D-B-tree, R-tree,

R+-tree, R*-tree, X-tree, …

Page 7: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

7

High-Dimensional (HD) Indexing Applications

• Multimedia databases (Images, Sounds, Movies)– Map multimedia object to a n-

dimensional point called feature vector

– Number of dimensions: typically 256 - 1000

– Indexing:• Actually index only feature vectors

• Data structures used:– same as for spatial databases (R-Trees, X-

trees)

– or, structures tailored to index specifically feature vectors(TV-Tree)

Page 8: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

8

HD Considerations (1)

• Main problem:– In general there is no total-ordering of

d-dimensional objects that preserves spatial proximity

• Data comes in two forms– N-dimensional points

– N-dimensional objects extended in space

• Objects can have rather complex shapes (extents)

• Typically abstract from the actual form and index some simpler shapes, such as Minimum Bounding Boxes (MBB) or n-dimensional hyper spheres

Page 9: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

9

HD Considerations (2)

• “Dimensionality curse”– As the number of dimensions

increases• performance tends to degrade (often

exponentially)

• Indexing structures become inefficient for certain kinds of queries

• Performance is often CPU-bound, not just I/O-bound as in traditional DBMS

Page 10: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

10

HD Queries Overview

• No standard algebra or query language

• The set of operators strongly depends on application domain

• Queries are usually expressed by an extension of SQL (e.g. abstract data types)

• Although there are no standards, some queries are common

Page 11: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

11

Multiattribute and Spatial Indexing of Multimedia Objects

• Spatial Databases: Queries involve regions that are represented as multidimensional objects.

Example: A rectangle in a 2-dimensional space involves four values: two points and two values for each point.

Access methods that index on multidimensional keys yield better performance for spatial queries.

• Multimedia Databases: Multimedia objects typically have several attributes that characterize them.

Example: Attributes of an image include coarseness, shape, color, etc.

Multimedia databases are also good candidates for multikey search structures.

Page 12: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

12

Measure of Similarity

A suitable measure of similarity between an image feature vector F and query vector Q is the weighted metric W:

where A is an nxn matrix which can be used to specify suitable weighting measures.

W F Q A F QT ( ) ( ) ,

Page 13: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

13

Similarity Based on Euclidean

Distance

matrix.identity theis where

),()(),(

A

QFAQFQFD T

F F F Q

D(F1 ,Q)

1 2 3

3

4

6

2

4

7

3

4

7

2

4

6

1 0 0

1 0 0

0 1 0

0 0 1

1

0

0

1

0 0 1

1 0 0

0 1 0

0 0 1

0

0

1

1

1 0 1

1 0 0

0 1 0

0 0 1

1

0

1

2

. similar to less is ),(),(

. similar to equally are and ),(),(

331

2121

QFQFDQFD

QFFQFDQFD

D(F2 ,Q)

D(F3 ,Q)

Page 14: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

14

Similarity Based on Euclidean Distance (cont.)

F1

F2

Feature 2

Feature 1

Points which lie at the same distance from the query point are all equally similar, e.g., F1 and F2.

F3

Q

Page 15: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

15

Similarity Based on Weighted Euclidean Distance

where A is the diagonal.

),(),( QFAQFQFD T

F F Q A

D(F1 ,Q)

1 2

4

5

7

3

5

8

3

5

7

1 0 0

0 1 0

0 0 2

1 0 0

1 0 0

0 1 0

0 0 2

1

0

0

1

0 0 1

1 0 0

0 1 0

0 0 2

0

0

1

2

Example:

D(F2 ,Q)

D(F1 ,Q) < D(F2 ,Q) F1 is more similar to Q

Page 16: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

16

How to determine the weights ?

A

0 0

0 0

0 0

Si

2: the variance of the i-

th feature measures.S2

2

S1

2

S3

2

The variance of the individual featuremeasures can be used as their weights.

Rationale: A feature with a larger variance is more discriminating.

Page 17: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

17

Query TypesQuerying in image DBMS is envisioned to

be iterative in nature:

• Vague Queries: Queries at the earlier stage can be very “loose”.

Retrieve images containing textures similar to this sample.

• K-nearest-neighbor-queries: The user specifies the number of close matches to the given query point.

Retrieve 10 images containing textures directionally similar to this sample

• Range queries: An interval is given for each dimension of the feature space and all the records which fall inside this hypercube are retrieved.

r is large r is small range query=> vague query => 3-nearest neighbor query

Q Q.Q. r

. .

.. . .

... . ..+ ++

Page 18: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

18

Indexing Multimedia Objects• Can’t we index multiple features

using a B+-tree ?

– B+-tree defines a linear order

– Similar objects (e.g., O1 and O2) can be far apart in the indexing order

• Why multidimensional indexing ?

– A multidimensional index defines a “spatial order”

– Conceptually similar objects are spatially near each other in the indexing order (e.g., O1 and O2)

Feature X

Fea

ture

Y

.O1

O2.

Page 19: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

19

Some Multidimensional Search Structures

• Space Filling Curves

• k-d Trees

• Multidimensional Tries

• Grid File

• Point-Quad Trees

• R Trees, R*, TV, SS

• D-Trees

• VA files

Page 20: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

20

Space Filling Curves

• Assume that each dimension is represented by a fixed bit width number

• Partition the universe with a grid

• Label each grid cell with a unique number called the curve value

• For points, store that number in a traditional one-dimensional index

• Objects can be handled through decomposition into multiple cells

Z-ordering Curve with 2

bits

Page 21: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

21

k-d Trees• k-d tree is a multidimensional binary search tree.

• Each node consists of a “record” and two pointers. The pointers are either null or point to another node.

• Nodes have levels and each level of the tree discriminates for one attribute.

• The partitioning of the space with respect to various attributes alternates between the various attributes of the n-dimensional search space.

Example: 2-D treeInput Sequence

A = (65, 50) B = (60, 70) C = (70, 60) D = (75, 25) E = (50, 90) F = (90, 65) G = (10, 30) H = (80, 85) I = (95, 75)

A(65, 50)

B(60, 70) C(70, 60)

G(10,30) E(50,90) D(75, 25)

F(90, 65)

H(80, 85) I(95, 75)

Discriminator

X

Y

X

Y

Page 22: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

22

k-d Tree: Search Algorithm

• Notations:

• Algorithm: Search for P(K1, ..., Kn)

Q := Root; /* Q will be used to navigate the tree */

While NOT DONE DO the following:

if Ki(P) = Ki(Q) for i = 1, ..., n then we have located the node and we are DONE

Otherwise if A = Disc(Q) and KA(P) < KA(Q)

then Q := Low(Q)

else Q := High(Q)

• Performance: O(logN), where N is the number of records

L

M N

(..., KA(L), ...)

M = Low(L)N = High(L)

Disc(L) : The discriminator at L’s levelKA(L) : The A-attribute value of LLow(L) : The left child of LHigh(L) : The right child of L

Page 23: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

23

Multidimensional Tries• Multidimensional tries, or k-d tries, are similar

to k-d tree except that they divide the embedding space.

• Each split evenly divides a region

Example: Construction of a 2D tries

X>75

X<=50 X>50

Y>50

X<=75

Y<=50

X<=75 X>75

Y>25

A(65,50)

X<=75 Y>75

X<=62.5 X>62.5

B(60,70) C(70,60)

X<=50 X>50

A(65, 50)

Insert A(65,50):

X<=50 X>50

Y<=50 Y>50

A(65,50) B(60, 70)

Insert B(60, 70):

Insert D(75, 25):

B(60, 70) C(70, 60)

X<=50 X>50

Y<=50 Y>50

X<=75 X>75

X<=62.5 X>62.5

Y<=75 Y>75

A(65,50)

Y<=25

D(75,25)

Insert C(70,60):

Partitioning of the space

10

20

30

40

50

60

70

10 20 30 40 50 60 70 80 90

1

4

3

2

7D(75,25)

B(60,70)

C(70, 60)

A(65,50)

5

X

Y

6

Page 24: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

24

Multidimensional Tries: Using Buckets

Disadvantage: The maximum level of decomposition depends on the minimum separation between two points.

A solution: Split a region only if it contains more than p points.

Page 25: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

25

Grid Files

Split Strategy: The partitioning is done with only one hyperplane, but the split extends to all the regions in the splitting direction

1. The directory is quite sparse.

2. Many adjacent directory entries may point to the same data block.

3. For partial-match and range queries, many directory entries, but only few data blocks, may have to be scanned.

linear scaleGrid directory

Data bucket

A B C D

D E F G

H I J J

K K L M

100

75

50

25

0 25 50 75 100

0 25 50 75 100

1 2 3 4

100

75

50

25

0

1

2

3

4

Page 26: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

26

Point-Quad Trees• Each node of a k-dimensional quad tree

partitions the object space into k quadrants.

• The partitioning is performed along all search dimensions and is data dependent, like k-d trees.

Example:

The quad treeD(35,85)

A(50,50)

E(25,25)

B(75,80)

C(90,65)

A

SE

SW

E

NW

D

NE

B

SE

SW NW

NE

C

To insert P(55, 75):

• Since XA< XP and YA < YP go to NE (i.e., B).

• Since XB > XP and YB > YP go to SW, which in this case is null.

Partitioning of the space

P

Page 27: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

27

Spatial Index Trees• We will talk about data normalized in

the range [0, 1] for all the dimensions.• Minimum Bounding Region (MBR)

refers to the smallest region (rectangle, circle) that encloses the entire shape of the objects or all the data points.

Page 28: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

28

R-tree

• R-trees are higher generalizations of B-trees.

• The nodes correspond to disk pages.

• All leaf nodes appear at the same level.

• Root and intermediate nodes corresponds to the smallest rectangle that encloses its child nodes, i.e., containing [r, <page pointer>] pairs.

• Leaf nodes contain pointers to the actual objects, i.e., containing [r, <RID>] pairs.

• A rectangle may be spatially contained in several nodes (e.g., J ), yet it can be associated with only one node.

A B C

D E F G H I J K L

D

F

E

G

H

IL

K

A

B

C

J

Page 29: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

29

• Hierarchy of nested d-dimensional intervals (boxes).

• Each node v corresponds to a disk page & d-dimensional interval, .

• Store MBB or MBR of n-dimensional object.

• Permits overlap of index entries.

• Index used as filter mechanism for query.

• Every node contains between m and M entries unless it is a root.

• The root node has at least 2 entries unless it is a leaf.

• Height-balanced.

Which of the above properties are similar to - trees ?

vI d

B

R-Trees

Page 30: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

30

R-tree: Insertion

• A new object is added to the appropriate leaf node.

• If insertion causes the leaf node to overflow, the node must be split, and the records distributed in the two leaf nodes.

– Minimizing the total area of the covering rectangles

– Minimizing the area common to the covering rectangles

• Splits are propagated up the tree (similar to B-tree).

Page 31: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

31

R-tree: Delete

• If a deletion causes a node to underflow, its nodes are reinserted (instead of being merged with adjacent nodes as in B-tree).

• There is no concept of adjacency in an R-tree.

Page 32: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

32

D-tree: Domain Decomposition

If the number of objects inside a domain exceeds a certain thresholds, the domain is split into two subdomains.

Example 1: Horizontal Split

Example 2: Vertical Split

A

B

CD

F

E

G

D

A subdomain

Split along longest dimension

Original domain

Split lineG F

E

DB

A

C

DB

A

C

G

E

F

A border object

A subdomain

Original domain

A

B

C

F

E

G

Page 33: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

33

D-tree: Split Examples

D-tree

Initial tree:D

Embedding Space

D

After 3insertions:

null

Data node

D

After 1st split:D1 D2

D1 D2

After 2nd split:D11 D2 D12

D2

D11

D12

Domainnode

null

null null

Page 34: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

34

D-tree: Split Example (continued)

D-tree Embedding Space

After 3rd

split:

D11

D121

D2

D122

D11 D2 D121 D122

After 4th

split:

Internal node

External node

D1 D2

D11 D121 D122 D21 D22

D122

D11 D21

D121 D22

D22.P

Page 35: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

35

D-tree: Range Queries

Note: A range query can be represented as a hypercube embedded in the search space.

Search Strategy:

• Retrieve the set, say S, of all subdomains which overlap with the query cube.

• For each subdomain, in S, which is not fully contained in the query cube, discard the objects falling outside the query cube.

Algorithm:

Search(D_tree_root, search_cube)

Current_node = D_tree_root

For each entry in Current_node, say (D, P), if D overlaps with search_cube, we do the following:

– If Current_node is an external node, retrieve the objects, in D.P, which fall within the overlap region.

– If Current_node is an internal node, call Search(D.P, search_cube).

Page 36: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

36

D-tree: Desirable Properties

• D-trees are balance

• The search path for an object is unique

No redundant searches.

• More splits occur in the denser regions of the search space.

Objects are evenly distributed among the data nodes.

• Similar objects are physically clustered in the same, or neighboring data nodes.

• Good performance is ensured regardless of the insertion order of the data.

Page 37: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

37

Content-Based Image Indexing

• Keyword Approach

– Problem: there is no commonly agreed-upon vocabulary for describing image properties.

• Computer Vision Techniques

– Problem: General image understanding and object recognition is beyond the capability of current computer vision technology.

• Image Analysis Techniques

– It is relatively easy to capture the primitive image properties such as

• prominent regions,

• their colors and shapes,

• and related layout and location information within images.

– These features can be used to index image data.

Page 38: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

38

Possible Features

• Edge

• Region

• Color

• Shape

• Location

• Size

• Texture

Page 39: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

39

EDGE

• Types of Edges – Step, Ramp, Spike and Roof.

• 3 stages in edge detection – Filtering : Image is passed

through a filter in order to remove noise.

– Differentiation : highlights the locations where intensity changes are significant.

– Detection

Page 40: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

40

Classes of edge detection schemes

• Prewit, Robert, Sobel, and Laplacian – 3x3 and 5x5 gradient operators

• Hueckel, Hartly and Haralick’s – surface fitting

• Canny - the derivatives of Gaussian

Page 42: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

42

Features Acquisition: Region Segmentation

• Group adjacent pixels with similar color properties into one region, and

• segment the pixels with distinct color properties into different regions.

Page 43: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

43

Definition of Segmentation

• All pixels must have the same ..

• All pixels must not differ by more than ..

• All pixels must not differ by more than T from the mean ..

• The standard deviation must small ..

Page 44: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

44

Simple Segmentation

• B(x, y) = 1 if T1 < f(x, y) < T2

0 otherwise

• Thresholds and Histogram

• Connected Component Algorithms– Recursive Algorithm– Sequential Algorithm

Page 45: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

45

Seed Segmentation

1. Compute the histogram

2. Smooth the histogram by averaging to remove small peaks

3. Identify candidates peaks and valleys

4. Detect good peaks by peakiness test

5. Segment the image using thresholds

6. Apply connected component algorithm

Page 46: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

46

Region Growing

• Split and Merge Algorithm

• Phagocyte Algorithm

• Likelihood Ratio Test

Page 47: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

47

Region Segmentation

• EDISON

• JSEG

Page 48: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

48

Color

• We can divide the color space into a small number of zones, each of which is clearly distinct with others for human eyes.

• Each of the zones is assigned a sequence number beginning from zero.

Notes: It is proven that human eyes are not very sensitive to colors. In fact, users only have a vague idea about the colors they want to specify.

Page 49: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

49

ShapeShape feature can be measured by properties: Circularity, major axis orientation, and Moment.

– Circularity:

Notes: The more circular the shape, the closer to one

the circularity.

– Major Axis Orientation:

– Moment : the first and the second

04

12

circularityarea

perimeter

circularityr

r

41

2 ( )

(2 )2

circularitya

a

4

4 4

2

2

( )

( )

circularitya

a

4 2

6

2

9

2

2

( )

( )

r

a

2a

a

a

420 oo 3500 norientatio

Page 50: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

50

Location

• The image is divided into sub-areas.

• Each sub-area is labeled with a number.

• The region location is represented by the number of the sub-area in which the centroid (gravity center) of the region is contained.

Note: When a user queries the database by visual contents, approximate feature values are used.

It is meaningless to use absolute feature values as indices.

0 1 2

3 4 5

6 7 8A

B

• Location of A is 4

• Location of B is 1

Page 51: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

51

Size• Total number of pixels occupied by

the region• The size range is divided into groups.• A region’s size is represented by the

corresponding group number.Example:

group number Size Range

1

2

3 2

4 2 3

5 3 4

6 4 5

7 5 6

8 6 7

9 7 8

10 8 9

14

12

12

A S A

A S A

A S A

A S A

A S A

A S A

A S A

A S A

A S A

A S A

sub sub

sub sub

sub sub

sub sub

sub sub

sub sub

sub sub

sub sub

sub sub

sub sub

Notes: Only the regions more than one-fourth of the sub-area are registered.

S: object size Asub: size of the sub-area

Page 52: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

52

Texture• Approach based on Statistics:

– angular second moment (energy, homogeneity or uniformity), entropy, correlation, inverse difference moment, contrast (inertia), variance, sum average, sum variance, difference variance, difference entropy, information measure of correlation I, information measure of correlation II, and maximal correlation coefficient.

• Approach based on human perception:– coarseness, contrast, directionality,

line-likeness, regularity and roughness – busyness, complexity and texture

strength – repetitiveness, orientation, and

complexity

Page 53: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

53

Image Indexing by contents

By applying image segmentation techniques, a set of regions are detected along with their locations, sizes, colors, texture and shapes.

These features can be used to index image data.

Page 54: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

54

Texture Areas

Texture areas and images with dominant high frequency components are beyond the capacity of image segmentation techniques.

Matching on the distribution of colors (i.e., color histograms) is a simple yet effective means for these areas.

Strategy: Dividing an image into sub-areas and creating a histogram for each of the sub-areas.

Note: the partitioning of the image is to capture locality information. We don’t want to match an image with a red balloon on top with an image with a red car in the bottom.

Page 55: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

55

Histograms• Gray-Level Histogram: It is a plot of the number

of pixels that assume each discrete value that the quantized image intensity can take.

• Color Histogram: It holds information on color distribution. It is a plot of the statistics of the R, G, B components in the 3-D color space.

R 3

B 3

G 2

R 3

B 3

G 2

Color ImageB

G

R

Color Histogram

3x3x2 = 18 bins

Image

Intensity

Gray-Level Histogram

Cou

nt36

18

10

white gray black

Page 56: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

56

We can use the largest, say 20, bins as the representative bins of the histogram.

these 20 bins form a chain in the 3-D color space.

If we can represent such chains using a numerical number, then we can index the color images using various tree structures.

• Connecting order: The representative bins are sorted in ascending order by their distance from the origin of the color space.

• Weighted Perimeter:

• Weighted Angle:

• Format of the index key:

Histograms (cont.)

iii

i dC

WP ,1

20

1 2

1

ii

i aC

WA

20

1 2

1

B

(0,1,1)

(2,3,0)

(3,2,3)

(6,2,0)

(8,2,6)

G

R0

di i 1, ai

Most histogram bins are sparsely populated,with only a small number of bins capturing themajority of pixel counts.

WP (10 bits) WA (10 bits)

Page 57: Indexing Techniques for Multimedia Databases  Multimedia Similarity  Search Structure  Image Indexing  Video Indexing

57

Color Correlogram