January 17, 2002 1

Unsupervised Segmentation of Color-TextureRegions in Image and Video

Yining Deng, B.S. Manjunath

Presented by

Chen-hsiu Huang

January 17, 2002

January 17, 2002 2

Abstract

In this paper, a new method for unsupervised segmentation ofcolor-texture regions in images and video is presented. This method,which refered to as JSEG, consists of two independent steps:

1. Color Quantization

Colors in the image are quantized to several representativeclasses that can be used to dierentiate regions in the image.

2. Spatial Segmentation

A region growing method is then used to segment the image anda similar approach is applied to video sequences. Besides, anadditional region tracking scheme is embedded into the regiongrowing process.

January 17, 2002 3

Image Homogeneity

In order to identify image homogeneity, the following assumptionsabout the image are made:

1. Each image contains a set of approximately homogeneouscolor-texture regions.

2. The color information in each image region can be represented bya set of few quantized colors.

3. The colors between two neighboring regions are distinguishable basic assumption in any color image segmentation algorithm.

January 17, 2002 4

Segmentation Example

Online demo can be found at http://maya.ece.ucsb.edu/JSEG/

Original 352x240 MPEG-1 video

frame

Segmentation with Pquant = 100,

Pscal = automatic, Pmerge = 0.4

January 17, 2002 5

Color Quantization

First, colors in the image are quantized to several representativeclasses that can be used to dierentiate regions in the image.This is performed in the color space without considering thespatial distributions of the colors.

Then, the image pixel values are replaced by their correspondingcolor class labels, thus forming a class-map of the image.

In the second stage, spatial segmentation is performed directlyon this class-map without considering the corresponding pixelcolor similarity.

January 17, 2002 6

Spatial Segmentation

Introduce a new criterion for image segmentation. This criterioninvolves minimizing a cost associated with the partitioning of theimage based on pixel labels.

A practical algorithm is suggested toward achieving thissegmentation objective. The notion of J-images is introduced.J-images correspond to measurements of local homogeneities atdierent scales, which can indicate potential boundary locations.

A spatial segmentation algorithm is then described, which growsregions from seed areas of the J-images to achieve the nalsegmentation.

January 17, 2002 7

Schematic of the JSEG algorithm

January 17, 2002 8

Color-maps After Quantization

Following quantization, the quantized colors are as-signed labels. Acolor class is the set of image pixelsquantized to the same color. Theimage pixel colors are replaced by their corresponding color classlabels. The newly constructed image of labels is called a class-map.

Examples of a class-map are shown:

January 17, 2002 9

Criterion for Good Segmentation (1/2)

Let Z be the set if all N data points in a class-map. LetZ = (z, y), z Z, and m be the mean,

m =1N

zZz

Suppose Z is classied into C classes, Zi, i = 1, ..., C. Let mi bethe mean of the Ni data points of class Zi,

mi =1Ni

zZiz

January 17, 2002 10

LetST =

zZz m2

and

SW =C

i=1

Si =C

i = 1

zZz mi2

SW is the total variance of points belonging to the same class.Dene:

J =ST SW

SW

January 17, 2002 11

Criterion for GOOD Segmentation (2/2)

For the case of an imageconsisting of severalhomogeneous color regions,the color classes are moreseparated from each otherand the value of J is large.

On the other hand, if allcolor classes are uniformly

distributed over the entireimage, J tends to be small.

January 17, 2002 12

Average J Notation

Now, lets recalculate J over each segmented regioninstead of theentire class-map and dene the average J by

J =1N

k

MkJk

We propose J as the criterion to be minimized over allpossibleways of segmenting the image given the number of regions.

A better segmentation tends to have a lower value of J . If thesegmentation is good, each segmented region contains a fewuniformly distributed color class labels and the resulting J valuefor that region is small. Therefore the overall J is small.

January 17, 2002 13

J-Image Introduced

Observe J , if applied to a local area ofthe class-map, is also agood indicator of whether that areais in the region interiors ornear region boundaries. We can now think of constructing animage whose pixel values correspond to these J values calculatedover small windows centered at the pixels

The higher the local J value is, the more likely that thecorresponding pixel is near a region boundary. The J-image islike a 3D terrain map containing valleys and hills that actuallyrepresent the region interiors and region boundaries, respectively.

January 17, 2002 14

Spatial Segmentation Algorithm

The size of the local windowdetermines the size of imageregions that can be detected.Windows of small size areusefulin localizing theintensity/color edges, whilelarge windows are useful fordetecting texture boundaries.

The characteristics of theJ-images allow us to use aregion-growing method to

segment the image.

From scale 1, the windowsize is doubled each time toobtain the next larger scale.

January 17, 2002 15

Flow-chart of Segmentation

January 17, 2002 16

Consider the original image as one initial region. The algorithmstarts the segmentation of the image at a coarse initial scale.Then, it repeats the same process on the newly segmentedregions atthe next ner scale.

Region growing consists of determining the seed points andgrowing from those seed locations. Region growing is followed bya region merging operation to give the nal segmented image.

The user species the number of scales needed for the image,which aects how detail the segmentation will be.

January 17, 2002 17

Segmentation Process

January 17, 2002 18

Seed Determination

A set of initial seed areas are determined to be the basis for regiongrowing. These areas correspond to minima of local J values.

January 17, 2002 19

Heuristic Measure

Calculate the average and the standard deviation of the local Jvalues in the region, denoted by J and TJ , respectively.

Set a threshold TJ at

TJ = J + aJ

Pixels with local J values less than TJ are considered ascandidate seed points. Connect the candidate seed points basedon the 4-connectivity and obtain candidate seed areas.

If a candidate seed area has a size larger than the minimum sizelisted in previous page, it is determined to be a seed.

January 17, 2002 20

Region Merge

Oversegmented regions are merged based on their color similarityand the distance between two color histograms i and j iscalculated by

DCH(i, j) = Pi PjWhere P denotes the color histogram vector.

January 17, 2002 21

Spatiotemporal Segmentation Scheme

The goal is to decompose the video into a set of objects in thespatiotemporal domain. Each object contains a homogenouscolor-texture pattern.

It can be seen that the overall approach for each video frame issimilar to the image segmentation work with the exception ofseed tracking and post-processing procedures.

The segmentation is performed on a group of consecutive frames.The number of frames, P, in each group can be set equal to thevideo shot length.

January 17, 2002 22

Seed Tracking

The seeds in the rst frame are assigned as initial objects. For each seed in the current frame, if it overlaps with an object

from the previous frame, it is assigned to that object; Else a newobject is created starting from this seed.

If a seed overlaps with more than one object, these objects aremerged.

Repeat Steps 2 and 3 for all the frames in the group. A minimum time duration is set for the objects. Very

short-length objects are discarded.

January 17, 2002 23

Jt Notation (1/3)

Sometimes, false merges occur when two seeds of neighboringregions overlap with each other across the frames due to objectmotion.

For two pixels at the same spatial location but in two consecutiveframes, a modied measure J can be used to detect if their localneighbors have similar color distributions. This new measure isdenoted as Jt

mi =1Ni

zZitz

January 17, 2002 24

Jt Notation (2/3)

LetST =

zZtz m2

and

SW =C

i=1

Si =C

i=1

zZitz mi2

The measure Jt is dened as

Jt =ST SW

SW

January 17, 2002 25

Jt Notation (3/3)

When a region and its surroundings are static, the Jt values aresmall for all the points. When a region undergoes motion,the Jtvalue will become large when two points at the same spatiallocations in the two frames do not belong to the same region.

Jt is calculated for each point in the seeds. A threshold isset suchthat only points with small Jt values are used for tracking andthe rest are discarded.

After seed tracking, seed growing is performed on individualframes to obtainthe nal segmentation results.

January 17, 2002 26

Parameter Selection

Threshold for the color quantization process. It determines theminimum distance between two quantized colors

The number of scales desired for the image. The last parameter is the threshold for region merging.

January 17, 2002 27

Conclusion

A new approach called JSEG is presented for the fullyunsupervised segmentation of color-texture regions in images andvideo.

A criterion for good segmentation is proposed. Applying thecriterion to local image windows results in J-images, which canbe segmented using a multiscale region growing method.

For video, region tracking is embedded into segmentation.Results show the robustness of the algorithm on real imagesandvideo.