RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES Mingming Lu, Qiyu Zhang, Wei-Hung Cheng, Cheng-Chang Lu Department

RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR

SEGMENTATION AND DIMENSION REDUCTION OF FEATURES

Mingming Lu, Qiyu Zhang, Wei-Hung Cheng, Cheng-Chang Lu

Department of Computer Science

Kent State University

September, 2009 Kent State University 2

Data Mining and Knowledge Management

• Processing multimedia objects

• Defining and extracting features

• Feature dimension reduction

• Multimedia data retrieval

• Knowledge representation and management


Current Tasks

• Off-line data training– Segment images – batch mode– Find region of interest (ROI)– Interface with feature extraction and analysis– Feature domain processing


Current Tasks (cont.)

• Users Interfaces– Reading user-input images– Segmentation– Find ROI– Feature extraction of ROI– Compare with trained data in repository– Return data (images) satisfying certain criteria


Data Training

Image &Feature DataRepository

Segmentation Finding ROI Interface Feature ExtractDimension Reduction

Sending Images for Processing Store Feature Data back

Image Domain Feature Domain


Image Domain Procesisng

• Segmentation – Color VQ, Texture based image segmentation

• Find ROI – ROI occupies large area– ROI locates near the image center– ROI contains homogenous texture


Color-Texture SegmentationApplications

• Identify Regions of Interest (ROI) in a scene

• Image classification

• Image annotation

• Object based image and video coding


Color-Texture SegmentationCurrent Limitations

• Many existing techniques work well on homogeneous color regions, while natural scenes are rich in color and texture.

• Many texture segmentation algorithms require the estimation of texture model parameters, which is a difficult problem and often requires a good homogeneous region for robust estimation.


Color-Texture SegmentationAdvantage of Color VQ and Texture

based segmentation • Does not attempt to estimate a specific

model for a texture region.

• Tests for the homogeneity of a given color-texture pattern, which is computationally more feasible than estimation of model parameters.


Color-Texture SegmentationTwo-Step Process

• Color Quantization– Performed in the color space without

consideration of spatial distribution of colors.– Label each pixel with a quantized color to form

a class-map.

• Spatial Segmentation– Performed on the class-map


Color-Texture SegmentationColor Quantization

• Use Peer Group Filtering

• As a result, coarse quantization can be obtained while preserving the color information in the original images.

• Usually 10-20 colors are needed in the images of natural scenes.


Color-Texture SegmentationCriteria for Good Segmentation

WWT

W

C

i

C

i ZziiW

ZzT

SSSJ

S

mzSS

mzS

i

/)(

Define

class. same the tobelonging points of variance total theis

and

Let

1 1

2

2


Color-Texture Segmentation-A Criterion for Good Segmentation

• When the color classes are more separated from each other, J is getting larger.

• If all color classes are uniformly distributed over the entire image, J tends to be small.


Color-Texture SegmentationA Criterion for Good Segmentation

• Now let us recalculate J over each segmented region instead of the entire class-map and define the average by

• A segmentation which can minimize J is considered a good segmentation.

k

Kk JMN

J1


Color-Texture Segmentation-Spatial Segmentation

• Seed Determination

• Seed Growing

• Region Merge


Color-Texture Segmentation-Spatial Segmentation


ROI Determination

• Find ROI – Mechanism– Pixel closer to the center contributes more

weight to the region it belongs to.

– Region with more pixels tends to get higher weight

dw

1


Results of Image Domain Processing

• Results of Color Quantization

• Results of Finding ROI


March, 2004 Kent State University 19

V1 = 500, V2 = 1, V3 = 0.5 AutoV1 = 500, V2 = 1, V3 = 0.5



V1 = 500, V2 = 1, V3 = 0.5 Auto



Interface with Feature Domain

• Find the rectangle circumscribing the ROI

• Store its coordinate information into to a temporary file for feature domain’s use.


Feature Domain(Overview)• Two Stages:

– Feature Extraction

– Dimension Reduction (DR)

Image &Feature DataRepository

Interface

Store Feature Data back

Image Domain Feature ExtractDimension Reduction

Feature Domain


Implementations

• Acquire ROI information from the image domain

• Extract features based on Gabor Filter and color histogram on HSV space

• Integrate two feature spaces

• Reduce the high feature dimensions to a very low number


Implementations (cont.)

• Calculate the similarity measurement between the query object and the objects in the image repository

• Search the similar images in the repository based on similarity index

• Output the corresponding retrieval images

• Knowledge extraction


Feature Extraction Algorithm

• Gabor Filter Feature– One of the most important wavelets with multi-

scale and multi-resolution – Mainly reflect texture information

• Color histogram on HSV space– Provide color features


Gabor Filter Concept

• A complete but non-orthogonal basis wavelet set

• A significant aspect: localized frequency description – composed of space information


Gabor(cont.)

• A two dimensional Gabor function g(x, y) and its Fourier transform G(u, v) can be written as:

jWx

yxyxg

yxyx

22

1exp

2

1),(

2

2

2

2

2

2

2

2)(

2

1exp),(

vu

vWuvuG

yvxu andwhere 2/1,2/1


Gabor(cont.)

• Let g(x, y) be the mother Gabor wavelet, then this self-similar filter dictionary can be obtained by appropriate dilations and rotations of g(x, y) through the generating function

number. scale themeans

ns.orientatio ofnumber total theis and , where

)cossin(),sincos(

int,,1),,(),(

m

Knππ/θ

yxayandyxax

nmayxGayxgmm

mmn


Color Histogram in HSV Space

• HSV color space includes – Hue (H)– Saturation (S)– Value (V or Lightness)

• Only consider Hue and saturation information, since the lightness of pictures is very sensitive to the surrounding conditions.


HSV space Figure


HSV space bands

• Design bands in the HSV space– 8 hue bands – 4 saturation bands, – Total 32 sub-spaces

• Compute color histogram feature in each sub-space to form 32 feature dimensions eventually


Feature Integration

• Normalize both Gabor filter and HSV color histogram features

• Set a weight factor to balance two feature spaces. Usually Gabor filter features will have the bigger weight value.


DR Algorithm

• Disadvantages in the high dimension space– The computational complexity arise sharply– The database indexing becomes difficult

• Principal Component Analysis (PCA) – PCA seeks to reduce the dimension of the data

by finding a few orthogonal linear combinations (Principal Component “PC”)


DR implementation

• Original feature dimensions– Gabor filter features: 6*5*2 = 60– HSV color histogram features: 4*8 = 32– Total dimensions: 92

• Feature dimensions after DR– 10 ~15 dimensions


Simulation Results in the Feature Domain

• We randomly select 11 query pictures as the test samples in this report.

• At each query time, at most 14 retrieval pictures are retrieved.

• The minimum square error method is served as the similarity measurement.

• The value in the tables as below means the positive pictures out of the 14 retrieval pictures.


Performance between different feature extraction techniques

• the integration of Gabor Filter and HSV color Histogram

gains the better performance. • See pictures in detail. Click here

Query pic#

1 2 3 4 5 6 7 8 9 10 11

Gabor 6 7 7 4 12 1 1 2 4 3 2

HSV 8 2 9 1 2 3 1 1 4 2 3

Integrated 10 5 11 4 12 3 3 2 5 2 4


Performance between with and without DR applied

• The performance after DR applied slightly degrades on average in comparison to the results before DR takes on

stage • See pictures in detail. Click here

Query pic#

1 2 3 4 5 6 7 8 9 10

11

Integrated

10 5 11 4 12 3 3 2 5 2 4

DR 9 6 5 5 12 2 1 1 4 3 2


More Simulations

• Performance between different weight used

• Performance between different dimensions retained after DR


Final Integration Results

• Simulation results when both the image domain and the feature domain are used

• See the detail pictures, Click here


Integration

• UAV media capture and analysis

• WWW based media analysis

• Vehicle based media capture and analysis


Future ResearchExtended to video objects

• Object based video coding

• Non-object based video coding

• Video indexing

• Knowledge extraction and management


Future ResearchData Fusion

• Multimodality medical imaging

• CT – Structural information

• PET – Functional information

• Fusion

• Knowledge management

Documents

RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES Mingming Lu, Qiyu Zhang, Wei-Hung Cheng, Cheng-Chang Lu Department