Vision Based Real Time Finger Counter for Hand Gesture Recognition

ISSN: 2277-4629 (Online) | ISSN: 2250-1827 (Print) CPMR-IJT Vol. 2, No. 2, December 2012

www.cpmr.org.in CPMR-IJT: International Journal of Technology 1

Vision Based Real Time Finger Counter forHand Gesture Recognition

S. Nagarajan* Dr. T. S. Subashini**

Dr. V. Ramalingam***

*, **, *** Department of Computer Science and Engineering, Annamalai University, Tamilnadu, India

ABSTRACTSign Language is a natural way of communicationfor deaf and mute people around the world. Theyuse different hand gestures to communicate withnormal people everywhere. In this paper, a real timefinger counter is proposed for counting the numberof fingers (1-5) shown in a hand gesture. This systeminvolves five phases namely input video capturing,preprocessing, hand region segmentation, featureextraction and recognition. In the first phase, thegesture video is captured using a low cost USBcamera in real time. In the second phase, the imageframes of the captured video are resized for furtherprocessing. The skin colour of hand region isdetected using HSV colour space and morphologicaloperations are performed in the third phase. In thefeature extraction phase, the convex hull method isused to detect the boundary points of the segmentedbinary hand image and the vertices of the convexpolygon are determined as the convexity defects i.e.finger tips. Finally, the recognition phase recognizesthe gesture number based on the number of convexitydefects present in the convex polygon. Theexperimental results show that this technique givessatisfactory recognition rate.

Keywords: Convex hull, Convexity defects, Featureextraction, HSV colour space, Skin coloursegmentation, Sign Language recognition

I. INTRODUCTIONHand gesture is a kind of non-verbal communicationused by the deaf people. They express different typesof emotions, orders and even cardinal information byhand gestures only. Sometimes, they can be the onlyway of communication for the mute persons (signlanguage) and police’s traffic co-ordination in theabsence of traffic lights. Every culture follows its owngestures which are not followed universally. Handgestures provide a separate complimentary modality tospeech for expressing one’s messages. Hence, a visualinterface can be developed for natural interactionbetween human and computers using hand gestures. Thekey problem in hand gesture interaction is to make thecomputers to recognize the hand gestures.

The existing approaches for hand gesturerecognition use either data gloves or Computer Vision.The former acts as a Sensor for digitizing hand and fingermotions into multi-parametric data. The other sensorscollect the hand configuration and movement. However,these devices are costlier and it needs skilful training



finger tips in [9]. Convex hull algorithm in [10] detectsthe boundary and centre of co-ordinate of hand imagefor real time recognition of hand gestures.

Recognition of sign language numbers using Thinningof segmented image is proposed in [11]. Image basednumbers of Persian Sign Language are recognized usingThinning method in [12]. [13] presents the convex hullalgorithm for detecting the boundary points of a 2Dobject. Various robust and efficient hand segmentationalgorithms are elaborated in [14]. Different imagemorphological operations are referred from [15].

III. PROPOSED METHODOLOGYThe experimental setup of finger counter requires a lowcost web cam and preferably a plain background withblack or green screen. It does not require any specialequipments such as colour data gloves. The systemtolerates minor camera motion including vibration.

The proposed work is shown in Fig.1

1. The gesture video is captured frame by frame usinga webcam as a first step.

2. The captured RGB image is resized into 300 X200 pixels in the pre-processing stage to improvethe processing speed and to reduce the calculationtime.

3. Then the image is converted from RGB to HSVcolour space [16].

4. The skin colour [18] of the hand is detected fromHSV image by applying a threshold value (less than18 or greater than 175) for hue component. Herethreshold of H is selected as 15.

5. Since the segmented binary hand image containsnoise blobs, any small blobs [17] which are lessthan 100 pixels are removed to filter the image.

6. Convex hull method detects boundary points andcentroid of hand.

7. By keeping the largest distance between centrepoint and the boundary point as radius, the ROI isdetected with a bounding box.

8. The vertices of the convex polygon are found outto identify the finger tips or convexity defects.

from the users. In contrast, Computer vision basedtechniques require only a camera for making a naturalinteraction between humans and computers without useof any special hardware equipments. These systemsprovide artificial vision rather than biological vision. Thebigger challenge in these systems are they need to bebackground invariant, lighting insensitive, person andcamera independent to achieve real time performance.But such systems offer better accuracy and robustness.

The enormous applications of hand gesturerecognition include Sign language recognition, Visualinterface for human computer interaction (HCI),pervasive computing, medical image navigation, remotecontrol of video games. The main objective of theproposed work is to design a vision based interfacewhich aims to recognize the sign language numbers fromthe gestures by counting the number of fingers based onconvex hull approach.

The remaining part of this paper is organized asfollows. Section II provides details about the relatedwork in this area. Section III presents the keyobservations and methodology of this work. SectionIV submits the experimental results of proposed work.Section V gives the conclusions.

II. RELATED WORKThe work in [1] presents a new approach for handgesture analysis for digit recognition. [2] explains a novelvision based method for recognizing finger actions bydetecting the fingertips and analysing finger tipparameters. In [3], the authors have proposed a novelalgorithm for detecting finger tip extracted using Bayesianrule based skin colour segmentation. The proposedwork in [4] uses boundary tracing for identifying numberof opened fingers for ASL alphabets. [5] describes thevarious hand gesture recognition techniques. A newtechnique in [6] detects the hand, determines its centre,tracks the hand’s trajectory and analyses the handlocations.

The various pixel based skin colour detectiontechniques for segmenting hand region are compared in[7]. A survey of recent hand gesture recognition systemsis presented in [8]. Thinning algorithm is applied to detect



9. The number of fingers is counted based on thenumber of convexity defects present in the convexpolygon and the count is displayed.

IV. EXPERIMENTAL RESULTSThe proposed work is implemented in Microsoft VisualStudio 2010 with OpenCV 2.1 and five gestures aretested with 30 different people having different handsize under varying lighting conditions. Each person showsfive gestures representing numbers from 1 to 5 in frontof the camera.

Experiments show that the response time using theFinger Counter is comparable to the time for selecting akey in the keyboard as it does not require a database oftraining images and it requires virtually no training onthe user side, because the gestures which it recognizesare very simple and intuitive. The outputs at differentstages of finger counter are given in Fig.2 and theexperimental results are tabulated in Table I.

Table 1

Confusion Matrix for of Finger Counter

Number of Fingers counted

1 2 3 4 5

% of Recognition

1 30 0 0 0 0 100 2 1 29 0 0 0 96.6 3 0 2 28 0 0 93.3 4 0 0 2 28 0 93.3

Input Gesture number

5 0 0 0 0 30 100

The average rate of recognition of finger counter is96.64%.

The rectangle seen in the output of recognitionmodule represents the bounding box and two blue dotsrepresent the convexity defects which are finger tips.The convexity defects are the outstanding areas of themask disturbing the continuity of the correspondingconvex hull. They are commonly used in gesturerecognition in order to identify the potential fingersegments. The start and end points of the convex hullwon’t be counted as defects. Hence the number of fingerscounted is given by the formula,

Number of Fingers counted=Number of convexitydefects-2

a. Original RGB image b. HSV image

c. Hue component d. Skin detected

Fig.1 Methodology of Finger Counter

→Hand Gesture

Capture input image from video

Resize the image frame to 300 X 200

Convert the RGB image into HSV colour space

Binarise the hand image by thresholding &remove noise

Detect boundary points and centroid of handusing convex hull

Extract the Region Of Interest

Find the Convexity defects (finger tips)

Recognize the number of fingers

Finger Count

→→

→→

→→

→→



e. Filtered image f. Convex hull

g. Convexity defects GUI of finger counter (Finger tips)

Fig. 2 Outputs of Finger Counter at different stages

V. CONCLUSIONSIn this work, we have focused mainly on hand gesturerecognition with a limited set of gestures presenting oneup to five fingers of the user’s hand. The number ofdefects are used to count fingers ratio but the possibilitiesare unlimited. In future, this can be extended to checkthe angle relationship between them, their relative lengthand so on. By requiring users to make a small numberof hand poses and hold their hands parallel to the imageplane, the system is able to make reliable recognition inreal time.

The main drawbacks of this proposed work arehand pose orientation and detection of convexity defectsin complex background if any other skin colour objectis present near to the user’s hand. These drawbackscan be overcome in real time hand gesture recognitionsystem with automatic orientation mechanism in future.

VI. REFERENCES[1] Ahmed BEN JMAA, Walid MAHDI, Yousra

BEN JEMAA and Abdelmajid BENHMADOU, “A new approach for digitrecognition based on hand gesture analysis”,

International Journal of Computer Scienceand Information Security (IJCSIS), Vol.2,No.1, 2009.

[2] Daeho Lee and SeungGwan Lee, “Vision-BasedFinger Action Recognition by Angle Detectionand Contour Analysis”, ETRI Journal, Vol.33,No.3, 2011.

[3] M.K. Bhuyan, Debanga Raj Neog and MithunKumar Kar, “Fingertip detection for Hand poserecognition”, International Journal ofComputer Science and Engineering (IJCSE),Vol.4, No.3, 2012.

[4] Ravikiran J, Kavi Mahesh, Suhas Mahishi,Dheeraj R, Sudheender S, Nitin V Pujari”,Finger Detection for Sign LanguageRecognition”, Proceedings of theInternational Multi Conference of Engineersand Computer Scientists (IMECS), Vol I,2009.

[5] Ginu Thomas, “A Review of Various HandGesture Recognition Techniques”, VSRDInternational Journal of Electrical,Electronics and CommunicationEngineering (VSRD-IJEECE), Vol.1(7), pp374-383, 2011.

[6] Mohamed Alsheakhali, Ahmed Skaik,Mohammed Aldahdouh, Mahmoud Alhelou,“Hand Gesture Recognition System”, TheIslamic University of Gaza, Palestine, 2011.

[7] Vladimir Vezhnevets, Vassili Sazonov, AllaAndreeva, “A Survey on Pixel- Based SkinColour Detection Techniques”, Graphics andMedia Laboratory, Mascow State University,Russia.

[8] Rafiqul Zaman Khan and Noor AdnanIbraheem, “Hand Gesture Recognition : ALiterature Survey”, International Journal ofArtificial Intelligence & Applications (IJAIA),Vol.3,



[9] Ahmad Yahya Dawod, Junaidi Abdullah, Md.Jahangir Alam, “Fingertips detection from colourimage with complex background”, The 3rdInternational Conference on Machine Vision(ICMV 2010), 2011.

[10] Xianghua Li, Jun-ho An, Jin-hong Min andKwang-Seok Hong, “Hand GestureRecognition by Stereo Camera using theThinning Method”, IEEE, 2011.

[11] Rajeshree Rokade, Dharmpal Doye, ManeshKokare, “Hand Gesture Reccognition byThinning Method”, International Conferenceon Digital Image Processing, IEEEComputer Society, 2009.

[12] Alaa Barkoky, Nasrollah M. Charkari, “PersianSign Language Numbers Recognition usingThinning method”, International Journal ofMultimedia Technology (IJMT), Vol.2, No.1,pp 27-31, 2012.

[13] “Convex Hull”, A notes on Introduction toAlgorithms, Recitation 24, May 6, 2011.

[14] Archana S. Ghotkar & Gajanan K. Kharate,“Hand Segmentation Techniques to HandGesture Recognition for natural HumanComputer Interaction, International Journalof Human Computer Interaction (IJHCI),Vol.3, Issue(1), 2012.

[15] R.C.Gonzalaz and R.E.Woods, Digital ImageProcessing, 2nd ed., Pearson education, 2002.

[16] http://en.wikipedia.org/wiki/HSL_and_HSV

[17] http://www.shervinemami.info/blobs.html

[18] Ciarán Ó Conaire, Noel E. O’Connor and AlanF. Smeaton, “Detector adaptation bymaximising agreement between independentdata sources”, IEEE International Workshopon Object Tracking and ClassificationBeyond the Visible Spectrum, 2007.

Documents

Vision Based Real Time Finger Counter for Hand Gesture Recognition