The MPEG-7The MPEG-7 Color DescriptorsColor Descriptors

The MPEG-7The MPEG-7 Color DescriptorsColor Descriptors

ByBy Bassam kehailBassam kehail

120060647120060647

Outline• Introduction• Visual Descriptors (color, texture, shape)• Scalable Color Descriptor (CSD)• SCD extraction• GoF/GoP Color Descriptor• Dominant Color Descriptor (DCD)• DCD Extraction• Color Layout Descriptor (CLD)• CLD extraction• Color Structure Descriptor (CSD)• CSD extraction• CSD scaling• Summary

Introduction•Color is an important visual attribute for both human

vision

and computer processing. Various factor influenced the selection of these color

descriptors, these include:1. Their ability to characterize the perceptual color similarity, judged by

performance of the descriptors in matching images and video segments based on color characteristics.

2. Low complexity of the associated extraction and matching techniques.3. The sizes of the coded descriptions, which play an important role in

indexing and in transmission of the descriptors over bandwidth-limited networks.

4. The scalability and interoperability of the descriptors.

IntroductionVisual Descriptors

Color DescriptorsDominant ColorScalable ColorColor LayoutColor StructureGoF/GoP Color

Texture DescriptorsHomogeneous TextureTexture BrowsingEdge Histogram

Shape DescriptorsRegion ShapeContour Shape3D Shape

(Normative, basic, for localization)

LocalizationRegion LocatorSpatio-TemporalLocator Motion Descriptors for VideoCamera MotionMotion TrajectoryParametric MotionMotion Activity

OtherFace Recognition

Type the Color descriptors• The Color Space Descriptor: Allows a selection of a color space to be used in the

description. The associated Color Quantization Descriptor specifies the partitiong of the given color space into discrete bins. These two descriptors are used in conjunction with other color descriptors.

• The Dominant Color Descriptors:Allows specification of a small number of dominant color values as

well as their statistical properties such as distribution and variance. Its purpose is to provide an effective, compact and intuitive representation of colors present in a region or image.

• The Scalable Color Descriptors:Is derived from a color histogram defined in the Hue-Saturation-

Value color with fixed color space quantization. It uses a Hear transform coefficient encoding, allowing scalable representation of description, as well as complexity scalability of feature extraction and matching procedures.

• The Group of Frames or Group of Pictures Descriptor:

Is an extension of the scalable Color Descriptors to a group of frames in a video or a collection of pictures. This descriptors is based on aggregating the color properties of the individual images or video frames.

• The Color Structure Descriptor:Is also based on color histograms, but aims at identifying

localized color distributions using a small structuring window. To guarantee interoperability, the color structure descriptor is bound to the HMMD color space.

• The Color Layout Descriptor:Captures the spatial layout of the representative colors on

a grid superimposed on a region or image. Representation is based on coefficients of the discrete Cosine Transform. This is a very compact descriptor being highly efficient in fast browsing and search applications. It can be applied to still images as well we to video segments.

Color Descriptors

Constrained color spaces:->Scalable Color Descriptor uses HSV->Color Structure Descriptor uses HMMD

Color Descriptors

Dominant Color Scalable Color- HSV space

Color Structure-HMMD space

Color Layout-YCbCr space

GroupOfFrames/Pictures

• Color Space: - R, G, B- Y, Cr, Cb- H, S, V- Monochrome- Linear transformation of R, G, B- HMMD

• Color Space:The color Space Descriptors specifies a selection of a color space

to be used in another color descriptor, specifically, the dominant color descriptor.

The color spaces specified in the MPEG-7 are RGB, YCbCr, HSV, HMMD, Monochrome and Linear transformation matrix with reference to RGB. In addition, a flag is provided to indicate reference to a color primary and mapping to a standard reference white value.

The Color Space Descriptor defines the color components as continuous-value entities. For discrete representation, a quantization is necessary. The color Quantization descriptor specifies the number of quantization levels for each color component in the color space. A uniform quantization in each of the color components in a given color space is assumed.

• The RGB color space is one of the more popular color models. This space is defined as the unit cube in the Cartesian coordinate system. The YCbCr is a legacy color space of the precedent MPEG standards, MPEG-1/2/4. It is defined by a linear transformation of RGB color space as follows:

• Y = 0.299*R + 0.587*G + 0.114*B• Cb = -0.169*R – 0.331*G + 0.500*B• Cr = 0.500*R – 0.419*G – 0.081*B

For the Monochrome color representation, Y component alone in the YCrCb is used.

Scalable Color Descriptor (CSD)

• A color histogram in HSV color space

• Encoded by Haar TransformFeature vector: {NoCoef, NoBD, Coeff[..], CoeffSign[..]}

SCD extraction

to 4bits/bin

to 11bits/bi

nNbits/bin

(#bin<256)

GoF/GoP Color Descriptor

• Histograms Aggregation methods:– Average ..but sensitivity to outliers (lighting changes, occlusion, text overlays)– Median ..increased comp. complexity for sorting– Intersection ..differs: a “least common” color trait

viewpoint

•Extends Scalable Color Descriptor for a video segment or a group of pictures (joint color hist. is then possessed as CSD- Haar transform encoding)

Extraction

Dominant Color Descriptor (DCD)

• Clustering colors into a small number of representative colors (salient colors)

• F = { {ci, pi, vi}, s}• ci : Representative colors

• pi : Their percentages in the region

• vi : Color variances

• s : Spatial coherency

DCD Extraction (based on Lloyd gen. algorithm)

ci centroid of cluster ;

x(n) color vector at pixel;

v(n) perceptual weight for pixel .

+spatial coherency:Average number of connecting pixels of a dominant color using 3x3 masking window

H.V.P more sensitive to smooth regions

Color Layout Descriptor (CLD)

• Clustering the image into 64 (8x8) blocks

• Deriving the average color of each block (or using DCD)• Applying (8x8)DCT and encoding

• Efficient for– Sketch-based image retrieval– Content Filtering using image

indexing

…

…

.

.

...

. .

.

If the time domain data is smooth (with little variation

in data) then frequency domain data will make low frequency data larger and high frequency data smaller.

-> derived average colors are transformed into a series of coefficients by performing DCT (data in time

domain - > data in frequency domain).

-> A few low-frequency coefficients are selected using zigzag scanning and quantized to form a CLD (large quantization step in quantizing AC coef / small quantization

step in quantizing DC ). ->The color space adopted for CLD is YCrCb.

CLD extraction

F ={CoefPattern, YDCCoef,CbDCCoef,CrDCCoef,YACCoef, CbACCoef, CrACCoef}

Color Structure Descriptor (CSD)

• Scanning the image by an 8x8 struct. element

• Counting the number of blocks containing each color

• Generating a color histogram (HMMD/4CSQ operating points)

8 x 8 structuringelement

COLORBINC0

C1 +1

C2

C3 +1

C4

C5

C6

C7 +1

CSD extractionIf

Then sub sampling factor p is given by:

F = {colQuant, Values[m]}

CSD scaling

The End