Robust Hand Tracking with Refined CAMShift Based on Combination of Depth and Image Features

Robust Hand Tracking with Refined CAMShift Based on Combination of Depth and Image FeaturesWenhuan Cui, Wenmin Wang, and Hong Liu

International Conference on Robotics and Biomimetics, IEEE, 2012

2

Outline• Introduction • Related Work• Proposed Method• Experimental Results• Conclusion

3

Introduction

4

Introduction• Hand Tracking:

• Essential for HCI• Most researchers simplify the issue by restrictions:

• On users’ clothing

• On the scene complexity

• On hand motion

Zhou Ren, Junsong Yuan, , Jingjing Meng, M, and Zhengyou Zhang, "Robust Part-Based Hand Gesture Recognition Using Kinect Sensor", IEEE TRANSACTIONS ON MULTIMEDIA, AUGUST 2013

5

Introduction• In this paper:

• Propose a robust hand tracking method

• Focus on reducing restrictions

• Combining:

• Depth cues• Color cues• (Motion cues)

Refined CAMShift tracking

6

Related Work

Related work• Tracking:

7

[a] fingertip

‧Seed Point‧Predicted hand position

[b] hand

Geodesicdistance

GSP points

Neighbordepth

Related work• Difficulties:

8

[c]

[d]

-- (Red) : Side-modeㄧ (Blue) : Frontal-mode

Related work• [a] Hui Liang, Junsong Yuan, and Daniel Thalmann, "3D Fingertip and Palm Tracking

in Depth Image Sequences", Proceedings of the 20th ACM international conference on Multimedia, 2012

• [b]Chia-Ping Chen, Yu-Ting Chen, Ping-Han Lee, Yu-Pao Tsai, and Shawmin Lei, "Real-time Hand Tracking on Depth Images", IEEE Visual Communications and Image Processing (VCIP), 2011

• [c] Ziyong Feng, Shaojie Xu, Xin Zhang, Lianwen Jin, Zhichao Ye, and Weixin Yang, “Real-time Fingertip Tracking and Detection using Kinect Depth Sensor for a New Writing-in-the Air System”, Proceedings of the 4th International Conference on Internet Multimedia Computing and Service, 2012

• [d] Zhichao Ye, Xin Zhang, Lianwen Jin, Ziyong Feng, Shaojie Xu, "FINGER-WRITING-IN-THE-AIR SYSTEM USING KINECT SENSOR", IEEE International Conference on Multimedia and Expo Workshops (ICMEW), 2013

9

10

ProposedMethod

Flow Chart

11

Hand Detection Hand Tracking

12

Foreground Segmentation• Codebook model

• Codeword:

• Motion detection(Foreground):

K. Kim, T. H. Chalidabhongse, D. Harwood, and L. Davis. Real time foreground-background segmentation using code book model. Real-Time Imaging, 11:172–185, 2005.

Down-Sampled

13

Foreground Segmentation

14

Histogram-based Segmentation• Stretch ahead

• Depth histogram

• Stretch laterally

• X-projection histogram

Mask

Mask

15

Histogram-based Segmentation• Stretch ahead

• Depth histogram

depth

16

Histogram-based Segmentation• Stretch laterally

• X-projection histogramLower boundary

Upper boundary

j-th bin

x

17

Histogram-based Segmentation• Histogram Analysis

• Depth histogram & X-projection histogram

• Foothill algorithm:

18

Histogram-based Segmentationmax

max01

01

01 10 01 10

01

Depth histogram

X-projection histogram

000000111111011100000

19

Histogram-based Segmentation

Scaled x-mask

X-projection histogram

20

Skin Color Feature

‧

‧

‧

Mask

21

Integration of Features• Hand Detection:

skin

depth(stretch ahead)

X-projection(stretch laterally)

22

• Like mean-shift

• 1. Back projection• Choose an object → probability map → back projection

• 2.Mean-shift (frame-frame)

CAMShift

23

Refined CAMShift Tracking• Probability map:

• Weights:

• s1 : depth mask• s2 : x-mask

blob size < threshold

otherwise

skin depth(stretch ahead)

X-projection(stretch laterally)

24

• Ecliptic shape representation

Refined CAMShift Tracking

Aspect ratio:

Search window for the next frame:

25

• Blob refinement

Refined CAMShift Tracking

1. Choose proper reference line

2.

3. Reduce the l of the ellipse, untill a proper aspect ratio l/w is obtained.

26

• Aspect ratio based blob refinement

Refined CAMShift Tracking

27

• Tracking fast movementDetection +

Tracking

28

• Face & Hand Detection +

Tracking

29

ExperimentalResults

30

Experimental Results• Comparison of overall performance

[10] C. Shan, Y. Wei, T. Tan, F. Ojardias, ”Real Time Hand Tracking by Combining Particle Filtering and Mean Shift”, In: International Conference on Automatic Face and Gesture Recognition, 2004, pp. 669-674

‧Training: 4.8s / 10FPS

31

• A) Refined CAMShift with color cue

32

• B) Multi-cue CAMShift without refinement

33

• C) The proposed approach

34

Experimental Results• Video description experimental results

35

Conclusion

36

Conclusion• Focus on reducing restrictions

• Hand Segmentation:• Depth + Skin + (Motion)• Histogram analysis

• Hand tracking• CAMShift• Blob refinement