1

A Robust Method of Detecting Hand Gestures Using

Depth Sensors

Yan Wen, Chuanyan Hu, Guanghui Yu, Changbo Wang

Haptic Audio Visual Environments and Games (HAVE), 2012 IEEE International Workshop on

2

Outline

Introduction

Related Works

The Proposed Method

Experimental Results

Conclusion

3

Introduction

4

Introduction

In human-computer interaction(HCI) system, recognizing hand and finger gestures are significant. Medical system, computer games, and human-robot

Depth-sensing camera(Kinect, Xtion) add a dimension to increase accuracy.

Goal: detect hand gestures with color and depth information

5

Related Works

6

Related Works

Body [4][5] V.S. Hand

Hand Superiority: simple

Inferiority: small scale, low resolution

Strict condition: cluttered background, lighting variation

7

Related Works

Hand gesture recognition Only color[12]

Data glove[7]

Training process[9][10]

Earth Mover’s Distance(EMD)[11]

8

References [7] R. M. Satava, “Virtual reality surgical simulator,” Surgical Endoscopy, vol. 7, pp.

203–205, 1993.

[9] C. Keskin, F. Kirac, Y. Kara, and L. Akarun, “Real time hand pose estimation using depth sensors,” in Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on, nov. 2011.

[10] P. Doliotis, A. Stefan, C. McMurrough, D. Eckhard, and V. Athitsos, “Comparing gesture recognition accuracy using color and depth information,” in Proceedings of the 4th International Conference on Pervasive Technologies Related to Assistive Environments, ser. PETRA ’11. New York, NY, USA: ACM, 2011, pp. 20:1–20:7.

[11] Z. Ren, J. Yuan, and Z. Zhang, “Robust hand gesture recognition based on finger-earth mover’s distance with a commodity depth camera,” in Proceedings of the 19th ACM international conference on Multimedia, ser. MM ’11. New York, NY, USA: ACM, 2011, pp. 1093–1096.

[12] A. Argyros and M. Lourakis, “Real-time tracking of multiple skincolored objects with a possibly moving camera,” in Computer Vision -ECCV 2004, ser. Lecture Notes in Computer Science. Springer Berlin/ Heidelberg, 2004, vol. 3023, pp. 368–379.

9

The Proposed Method

10

The Proposed Method

Gesture Representation

Finger RecognitionFind convex hull Detect fingertip and direction

Hand SegmentationFind hands through color

Separate hands

by k-meansFind palm center

11

Hand Segmentation (I)

Find hands through color

Train skin-color[12], detect face[15], image filtering[16], color threshold

L*a*b color space b

Operate AND on the two images

Minimum depth (10cm)

12

Hand Segmentation (I)--Find hands through color

RGB image Depth image L*a*b color space where b = 2

L*a*b color space where b = 3

Skin color images after ANDoperation

Binary image of hand segmentation

13

Hand Segmentation (II)

Separate hands by k-means k=2

Assignment:

Update:

Threshold of distance between 2 clusters

14

Hand Segmentation (II)--Separate Hands By K-means

15

Hand Segmentation (III)

Find palm center Inscribed circle

Minimum inner distance

Maximum element of inner distances set

16

Finger Recognition (I)

Find convex hull Graham’s scan algorithm

P: the lowest y-coordinate

Sort in increasing order of angle

Point to point is left/right turn

Left-turn: O ; right-turn: X

17

Finger Recognition (II)

Detect fingertip and direction Fingers are long and narrow

Find an isosceles triangle with V V: Every vertex on the convex hull

Set a maximum threshold to the vertex angle

The direction vector is paralleled with the median length of an isosceles triangle

18

Finger Recognition

Blue point : cluster centroid Green point : palm centerRed points : fingertipsYellow curves : hand contourLong lines : finger directions Structures around the hand : convex hull

19

The Proposed Method

Gesture Representation

Finger RecognitionFind convex hull Detect fingertip and direction

Hand SegmentationFind hands through color

Separate hands

by k-meansFind palm center

20

Gesture Representation

All information about hands Palm center location

Finger number

Fingertips location

Finger direction vectors

Gestures Rock-paper-scissors game

Drag images

Grasping, releasing

21

Experimental Results

22

Experimental Results

Use Kinect as input of depth and color images

The detection successful rate can reach 95%.

No matter the hand is horizontally or vertically placed.

23

24

Experimental Results

25

Experimental Results--Shadow Puppetry

26

Conclusion

27

Conclusion

Present a new method to detect hands’ positions and gestures

NO training, NO database

Future works Set a threshold to the distance between the palm center

and the fingers

Add additional sensor devices to overcome no palm detection

Shadow Puppetry project