Proceedings of the 2012 International Conference on Machine Learning and Cybernetics, Xian, 15-17 July, 2012

HAND SEGMENTATION USING SKIN COLOR AND BACKGROUND INFORMATION

ISchool of Electronic Information Engineering Tianjin University, Tianjin 300072, China 2School of Electronic Engineering, Tianjin University of Technology and Education, Tianjin 3000222, China

E-MAIL: visunny@sina.com.jingpan23@gmail.com

Abstract: Precise hand segmentation is crucial for gesture-based

Human-Machine Interaction. Skin color based hand

segmentation using skin color models shows poor performance

in complex background where similar colors of the skin and

non-uniform illumination exist. We propose a new method for

hand segmentation by using an adaptive skin color model and

the background information around the hand. Firstly, our

method captures pixel values of the hand and the background

then converts them into YCbCr color space. Secondly, skin and background Gaussian models based on the color space of

CbCr are proposed. Lastly, these models are taken to segment the whole image respectively, and then required for the

intersection.

The main contribution of the paper is that the

background information is taken into account to split image in

reversed side to enhance the performance. Experimental

results show that our method outperforms the method that

uses the skin color model only.

Keywords: Hand segmentation; Color models; Background

information; Gaussian model

1. Introduction

Recognizing hand signs and tracking hand motion is crucial for human machine interface which has always been a hot topic in recent years. The purpose of hand signals recognizing and hand fingers tracking is to make the machine get our instructions in a non-contact way. O ther applications such as helping the disabled people communicate with the normal people also show the importance of the research in this area. Hand segmentation is the first and also the critical step for recognizing hand signs and tracking hand motion. The excellence of the hand segmentation affects the accuracy of its following applications in a straight way.

As experience suggests, human skin has a distinguishable color, which can be easily recognized by

978-1-4673-1487-9/12/$31.00 ©2012 IEEE

human [1]. Hence, skin color based hand segmentation has been heavily investigated for decades since it is simple and intuitive [2], [3], [4], [5]. Choosing appropriate color space and model of the distribution of skin color are two critical parts of hand segmentation. In [6], Y. P. Lew segmented the skin color in normalized RGB color space and modeled the distribution by Gaussian model. A combination of HSI and YCbCr color space has been used in [7], and edge detection method has also been used for the region boundaries detection in the same work. Comparison of two

chrominance models (single Gaussian model and Gaussian mixture model) and nine different chrominance spaces has been studied in [8]. Nine skin color classifiers (Bayesian classifier, three linear classifiers, three unimodal Gaussian classifiers, a Gaussian mixture classifier and multilayer perceptron classifier) were compared in [9] in different color space. In [4], A.Y. Dawod performed the skin color segmentation by using a free-form skin model instead of Gaussian model in YCbCr color space. S. L. Phung [10] proposed a skin color segmentation algorithm based on Bayesian skin color model by using skin and edge information in several color spaces (RGB,CIE,XYZ, HSV, YCbCr). All of these methods get rid of the background information which can split image in reversed side to enhance the performance. Yet, similar skin colors of the image cannot be classified correctly based on color models only in any color space. This means that non-skin pixel will be misclassified as skin-pixel because of the inclusion of these non-skin colors in the boundary box. Making full use of the background information can reduce the misclassification.

The primary goal of this paper is to make the usefulness of the background information known to other researchers in hand and other objects segmentation area. Thus we choose single Gaussian model to simulate the distribution of skin color. Experiments are performed in both Normalized RGB color space and YCbCr color space to avoid the interference of the color space.

1487

Proceedings of the 2012 International Conference on Machine Learning and Cybernetics, Xian, 15-17 July, 2012

The arrangement of this paper is organized as follows: section 2 is a brief introduction of the conversion of the two color space and the single Gaussian model. We interpret the proposed algorithm in section 3 and discuss the experiment results in section 4. Finally, conclusions are given in section

5.

2. Color space and Gaussian model

As discussed in the previous paragraph, which color space should be used and how to model the distribution of skin color are the two key problems the researchers should pay attention to. However, the primary goal of our method is to prove the usefulness of the background information around the hand in reducing the misclassification. Then, Normalized RGB and YCbCr color space are chosen due to its simplicity and extensive application.

2.1 Color space used for skin modeling

2.1.1, Normalized RGB

RGB is a convenient color model for computer graphic

because the human visual system works in a way that is similar to an RGB color space, when it was convenient to express color as a combination of three based colored rays (red, green and blue). RGB color space is widely used for processing image data.

RGB color space is not robust for color modeling since an RGB value cannot define the same color in different conditions with different illumination. Regarding of these

facts, Normalized RGB color space has been proposed. Normalized RGB skin color model has been proved to get better performance than RGB under different light conditions, but only in uniform illumination. Normalized RGB skin color model is considered to be more proper for hand skin, and can be easily obtained from the RGB values by a simple normalization processing:

R G B R =

R+G+B;G =

R+G+B;B =

R+G+B (1)

2.1.2. YCbCr

YCbCr has been considered to be better in describing the properties than RGB color space [4], [11]. The clustering characteristic for YCbCr is better than RGB [4]. YCbCr is used to separate out a luminance signal (Y) and two chrominance components (Cb and Cr). YCbCr can challenge various illumination conditions by discarding the signal Y, which not only improve the performance and also reduce the data dimension than RGB. The transform from

2.2 Single Gaussian model

The properties of skin color can be characterized by Gaussian distribution [12]. The single Gaussian model is one of the simplest models to model the distribution of the certain objects which is widely used in computer vision and pattern recognition. Gaussian distribution is given by the following formula:

[(X-I')' ]

f(x) =

1 e

-

(20-') �27r(/

(3) where f1 is the mean value of the samples and (j' the

variance value. U sing the Gaussian model to model skin color is

actually a process that matching each pixel of the image to

the model. If matched, we consider the pixel as a skin pixel, else we consider it background .The two parameters (m ands ) decide the structure of the Gaussian model, and

different learning mechanism of the two parameters directly influence the stability and accuracy of the model. A widely used method is training Gaussian model using thousands of sample images in various illumination and background

offiine [13], and the trained parameters are used to build the Gaussian model to characterize the skin color online. However, different people have different color, and the same people under different environment may show different color. Considering these differences, the offiine trained Gaussian model for skin color may be not suitable for characterizing all the people in various conditions. Thus we use an adaptive skin color model [4], which makes use of the center of the hand skin to calculate the two parameters of the Gaussian model and updated constantly without training offiine. Figure 1 show the results of the

two modeling approach.

1488

Proceedings of the 2012 International Conference on Machine Learning and Cybernetics, Xian, 15-17 July, 2012

a ROIimage

b results with ofHineGaussian model

c results with online Gaussian model

Figure 1. The comparison of the offline and online Gaussian model

As shown in Figure 1, the online Gaussian model shows more adaptability than the offline one in different images of different scenes under different illuminations. In the last scene in a, the online model can segment the hand completely, while the offline model cannot do.

3. Proposed method

The following algorithm introduces a new technique for segmenting hands based on the mixture of the background and skin color model. Figure 2 is the flowchart of hand segmentation procedure based on the mixture model. Detail description of the procedure is in the following text.

As the prerequisite, hand has been correctly detected. The Region of Interested (ROI) of the image is the detected region that contains hand, which has been extracted as the input image in our method. Figure 3 shows the example of the detected image and the ROI of the images.

Secondly, a region p. contains the skin color is

cropped in the center of the hand region; around the hand

N cropped regions (�, P" ... , PN) contains the background

color obtained randomly. Six directions of the models are built to make sure the skin color area is obtained definitely correct, and the background color areas correct as far as possible. Figure 4 presents the six direction models, where the rectangle in red contains the skin color definitely, and the rectangles in green contain most of the background color and some of the skin color.

detection Original image Ror

I .. ..

Sun'lvlng J)iscal"{l falsc Sclc(( Seleet kill I>n�kground

Bnckg70und background Croll crop hll.jlC

crOll linage lin age rn ndolllly crop image

1 Ilnckground Skin color

color 1lI0del llIodel

• • B,)ckgrollnd kin

segmentation "'gmentalion

I lulersecllon I ..

I'lnal resul,

Figure 2. The flowchart of the proposed method

a The original imlge b. The detected imoge c. The ROI imlge

Figure 3. Detected image and the cropped image of the whole image

Orientation 1 Orientation 2 Orientation 7

Orientation 4 Orientation 5 Orientation 6

Figure 4. Six direction models

It can be seen in Figure 4, some of the crop regions of the background is skin color actually because of the randomicity. At next step, we have to make sure whether the background color belongs to the background, or the

1489

Proceedings of the 2012 International Conference on Machine Learning and Cybernetics, Xian, 15-17 July, 2012

hand. The mean value (ME) and variance value (s ) of all

the crop regions are calculated to set up the Gaussian model. Each background crop region is used to segment the skin

original image is 320 X 240, and the size of the cropped

image is depended on the detected rectangles in the original image.

crop region with an automatic threshold interval r;. Pixel 4. Experiment results value in the skin crop region among r; set to 255 (white

pixels), others set to 0 (black pixels). If the proportion of the white parts in the obtained binary image exceeds a fixed threshold S , the corresponding background crop image is

considered the truth background and be accepted, otherwise, discarded. It should be pointed out that in order to ensure the color in the background is of single color as far as possible, the background crop region whose variance above

a certain value Vm is to be dropped.

Thirdly, the skin crop regions and the surviving background crop regions are taken to segment the cropped

image with the threshold � and r; respectively. The

segmentation results is (Ro' R], ... , RJ, where Ro is the

skin color segmentation result and ( R], R2, • • • , Rn) the

background color results.

Finally, all of the segmentation results are intersected to get the final result:

R = Ro 1fR, ... Rn Figure 5 shows parts of the segmentation results and

the final result. r-----------------------------,

/}r(> ' r� � , .

intersection

(Ji) R , "

�R, __ 1I"

;a1 R,

a partsoftbesegmentatioos b tbefinalr=l!s

Figure 5. Parts of the segmentation results and the final result

In our method, N = 50 ; Vrn = 0.15 ; S = 0.95

Interval r; of the background crop region depends on

S i which guarantees 90% of the pixel in the crop region is

in its Gaussian distribution. T,? [MFi S i MEi +s i] ' The

After describing the proposed method exhaustively, the results of the experiment will show in the following text. Figure 6 shows the comparison of the proposed method (we call it the skin and background based method, S&B) and the traditional skin color based method (skin) in Normalized

n b skin & S&Bresu1tsinRGB

a ROIimage n n c skin and S&B :results in YCbCr

Figure 6. Different results in RGB and YCbCr color space

RGB color space and YCbCr color space with the same data. It can be seen that the proposed method is far better than the traditional skin based method in both color space under the complicated environment with varying illumination.

We have tested our method on different images with varying illumination and complex background that contains the color very similar to the skin color. Figure 7 shows some results in YCbCr color space. Figure 7(a) shows the image is nearly blending with parts of the background under the dim light, and the edge of the hand is indistinct. Nevertheless, hand can be segmented clearly with our method shown in Fingure7(c).

In order to find out whether the number of the cropped regions, which we assumed containing the background information around the hand in the second step, has influence on the final results, we have compared the different parameters N , corresponding to the different

performance of the method in YCbCr color space, showed

in Figure 8. It can be seen that the performance of the proposed method is improved with the increasing of N in

a certain range. However, when N exceeds a certain value (40), the performance stays steady. It can be explained that N should be large enough to guarantee all of the useful

background information to be obtained. N should be smaller

1490

Proceedings of the 2012 International Conference on Machine Learning and Cybernetics, Xian, 15-17 July, 2012

on the image in which background is not very complex.

a The ROI images

r.n b segmented hands with skin color model

c segmented hands with S&B color model

Figure 7. Segmented hands by the proposed method

n n ROI image N=O N=lO

n PI PI N=20 N=40 N==80

Figure 8. Different results with different N

The essential part of the hand segmentation research is to find out the hand region as close as to the accurate one on the image immediately. We evaluate the accuracy rate

(AR) for the image with our method using the following equation.

AsnAL AR= ---

ASUAL (4)

where AS is the area of the segmented hand using our method, AL is the area of the accurate one of the same image in which the hand area has been labeled before. We can see that AR is less than 1 in any case, and the higher the

value of AR, the better the performance of the hand segmentation method. The evaluation of the results shows that the AR can reach to 90.36% by using our method on the

image that is difficult to segment, while 69.30% using the online skin model only.

5. Conclusions

In this paper, the usefulness of the background information in hand segmentation has been introduced. This method emphasizes the importance of the background

information that can be the supplement to the traditional skin color segmentation. The experimental results indicate that the background information is useful, especially prominent in waken the interference of the non-uniform illumination to the segmentation result, which can be extended to other object segmentation, for example, face.

In this paper, we use the simplest model (single Gaussian model) to characterize the skin color in norma­lized RGB color space and YCbCr color space respectively. Although the single Gaussian is unable to describe the correlation of color information in two or three channels, satisfying results are still obtained. In future work we mean

to use this method with Gaussian mixture model or other free-skin color model to obtain better performance.

Reference

[1] V. Vezhnevets, V. Sazonov, A Andreeva, "A Survey on Pixel-Based Skin Color Detection Techniques", Proc. Graphicon-2003, Pages: 85-92, Moscow, Russia, September 2003.

[2] Jie Yang, A Waibel, "A Real-time Face Tracker", Application of Computer Vision, Pittsburgh, Pages: 142-147, December 1996.

[3] Shu Mo, Shihai Cheng, Xiaofen Xing, "Hand Gesture Segmentation Based on Improved Kalman Filter and TSL Skin Color Model", International Conference on Multimedia Technology, China, Pages: 3543-3546, July 2011.

[4] AY. Dawod, lAbdullah, M.IAlam, "A New Method for Hand Segmentation Using Free-form Skin Color Model", International Conference on Advanced Computer Theory and Engineering, Cybetjaya, Malaysia, Pages: 562-566, September. 2010.

[5] M.l Jones, 1M. Rehg, "Statistical color models with application to skin detection", International Journal of Computer Vision, Volume: 46, Issue: 1, Pages: 81-96, 2002.

[6] Y. P. Lew, A R. Ramli, S. Y. Koay, "A hand segmentation scheme using clustering technique in

1491

Proceedings of the 2012 International Conference on Machine Learning and Cybernetics, Xian, 15-17 July, 2012

homogeneous background", Student Conference on Research and Development Proceedings, Malaysia, Pages: 305-358, November 2002.

[7] H. C. V. Lakshmi, S. Patikulakarni, "Segmentation Algorithm for Multiple Face Detection in Color Images with Skin Tone Regions using Color Spaces and Edge Detection Techniques", International Conference on Signal Acquisition and Processing, Pages: 162-166, February 2010.

[8] J. C. Terrillon, M. N. Shirazi, H. Fukamachi, "Comparative Performance of Different Skin Chrominance Models and Chrominance Spaces for the Automatic Detection of Human Faces in Color Images", Automatic Face and Gesture Recognition, Kyoto, Pages: 54-61, March 2000.

[9] Phung, S.L. "Skin segmentation using color pixel classification: analysis and comparison", Pattern Analysis and Machine Intelligence, Australia, Pages:

148-154, January 2005. [10] S. L. Phung, A. Bouzerdoum, D. Chai, "Skin

Segmentation Using Color and Edge Information", International Symposium on Signal Processing and Its Applications, Pages: 525-528, July 2003.

[11] Son Lam Phung, A. Bouzerdoum, "A Novel Skin Color Model in YCbCr Color Space and Its Application to Human Face Detection", International Conference on Image Processing, Australia, Pages: 289-292, 2002.

[12] M. H. Yang, N. Ahuja, "Gaussian Mixture Model For Human Skin Color and Its Applications In Image and Video Databases", SPIE, 1999.

[13] Ship eng Xie, Jing Pan, "Hand Detection Using Robust Color Correction and Gaussian Mixture Model", International Conference on Image and Graphics, Hefei, Pages: 553-557, August 2011.

1492