Particle Filter Object Tracking Based on SIFT-Gabor Region Covariance Matrices

Xinying Liu Yantai Vocational Institute

Yantai, Shandong Province, China, 264670 xinying.yt.liu@gmail.com

Abstract—Currently, object tracking is an important problem to computer vision community. It is usually performed in the context of higher-level applications aiming to accurately label and track target objects in frame sequences. However, video-based object tracking is very challenging, since the objects are easy to lose when illumination varies or occlusion occurs. To solve these problems, considering the SIFT and Gabor features perform robustly for objects representation, a novel method is proposed in which target model is constructed by SIFT-Gabor Region Covariance Matrices (SG-RCMs) and particle filter is used to track the object. In the tracking process, the target model is updated automatically according to the matching result between target model and candidate targets. Experimental results showed that the proposed approach tracks the object of which illumination and scale are drastically changing, effectively, accurately and robustly.

Keywords-Object tracking, particle filter, SIFT, Gabor, Region Covariance Matrices

I. INTRODUCTION

Object tracking is an interesting area within the field of computer vision. It is promoted with the increasing need for automated video analysis, the availability of high quality and low-cost video cameras, and the development of high-powered computers [1-2]. The use of object tracking is pertinent in the tasks of motion-based recognition [3], automated surveillance [4], video indexing [5], human-computer interaction [6], traffic monitoring [7], and vehicle navigation [8], etc.

Visual object tracking is really challenging, but in recent years, filter-based object trackers have proven to be very effective. The filter-based tracking methods could be divided into linear filtering and non-linear filtering. Among the linear filtering, Kalman filter is one of the most popular algorithms used in the computer vision community for tracking. T. Broida et al. [9] applied Kalman filter to estimation object motion in noise frames. R. Rosales et al. [10] used the extended Kalman filter (EKF) to recovery 3D trajectory for tracking multiple objects with 2D motion. However, Kalman filter does not work correctly when the systems is non-Gaussian.

The limitation above can be overcome by using non-linear filtering, i.e. particle filtering [11]. Particle filters are sequential Monte Carlo methods based on point mass representations of probability densities, which are applied to any state model, including the modeling of non-linear and non-Gaussian systems. Recently, particle filter has been extensively applied

to object tracking [12-13]. However, the tradition particle filter is based on color histogram of interesting regions, which leads it to miss a tracking when the illumination varies or occlusion occurs. The algorithm could also fail if the color distribution of the background is similar to the target. To solve this problem, P. Wu et al [14] combined color and SIFT feature to particle filter for object tracking problem because the SIFT feature invariabilities for illumination, scale and affine. Although the SIFT feature points are benefit for tracking, several problems remain. For example, the keypoints are scarcity when objects are smooth, which may lead the method sensitive to background noise interference. Later work applied a kernel particle filter for multiple-objects tracking based on Gabor feature region covariance matrices (RCMs) [15].

Considering the Gabor features is particularly suitable for image represent and exhibit strong characteristics of spatial locality, scale, and orientation selectivity, the SIFT and Gabor feature are combined to describe the target object in this paper. Inspired by the promising tracking results of RCMs [16-17], in this paper, SIFT and Gabor features are introduced into computing RCMs to significantly enhance the descriptiveness of target object. Hence, the proposed SIFT-Gabor-based RCMs (SG-RCMs) can achieve desirable tracking performance. The proposed method in this paper bears closest resemblance to the work of [15], but a number of improvements are made in the work.

The first contribution described in this paper is the fusion of SIFT and Gabor feature for object representation, to take the advantage of the two features. In the process of tracking, the label window may be much larger than the target when only using Gabor feature, and SIFT points are scarcity when objects are smooth. The fusion of the two features can enhance the tracking more accurate and robust.

The second difference is that the work in this paper is simpler and suffers less computing load. This work proposes a simpler and more efficient method to meet the need of real time application, as the original particle filter is applied instead of kernel particle filter in [15].

To the best of our knowledge, this is the first particle filter tracking method based on SIFT-Gabor Region Covariance Matrices that is able to track objects when illumination varies or occlusion occurs.

This paper is organized as follows. In Section II the framework of the proposed object tracking method is presented. The novel SIFT-Gabor-based RCMs algorithm is proposed in Section III and experimentally comparisons with tradition ___________________________________

978-1-4673-1332-2/12/$31.00 ©2012 IEEE

Particle filter, SIFT based particle filter and Gabor based particle filter are shown in Section IV. The conclusions are drawn in Section V.

II. FAMEWORK OF METHOD

The aim of this this paper is to accurately label and track target objects in frame sequences, despite the presence of background clutter, illumination varies or occlusion occurs. To achieve excellent performance, selecting the right features plays a critical role in the proposed tracker. Generally, the goal of feature generation is to select desirable property so that the target object can be well distinguished with other objects.

In recent years, particle filter-based object trackers have proven to be very effective in some tracking task. The traditional particle filter tracking weights each particle by consideration the Bhattacharyya coefficient of two corresponding color histogram. However, the color feature may be influence significantly with the brightness of the image changing. Moreover, the tracker may miss the target when the color distribution of the background is similar to the target. To solve these problems, target objects are descripted by the SFIF and Gabor features in this paper.

The SIFT features are invariabilities for illumination, scale and affine, and Gabor features is particularly suitable for image represent and exhibit strong characteristics of spatial locality, scale, and orientation selectivity. To combine the advantage of the SIFT and Gabor feature, both of these feature are embedded into the Region Covariance Matrices (RCMs), which presents several advantages as region descriptors and provides a natural way of fusing multiple features. The framework of the proposed algorithm for object tracking is show as Fig.1.

Fig.1 Framework of the proposed algorithm

III. SG-RCMS FOR OBJECT TRACKING

A. Patricle Filter Particle filter and its framework is extensively applied for

objects tracking to achieve a robust predicting and tracking method, it approximates the filtered posterior distribution by a set of weighted particles. Suppose that kx and ky are the state and observation respectively, then the object tracking problem can be transform to the approximation of the posterior distribution 1:( | )k kp x y is approximation, where

1: 1 2( , ,..., )k ky y y y� is the observation up to time k . The key idea of particle filter is to approximate the posterior distribution of the object state by a weighted particle set.

Given N particles 1: 1 1: 1 1{ , }i i Nk k ix w� � � at time 1k � ,

approximately distributed according to the distribution 1: 1 1: 1 1: 1( , | )i i

k k kp x w y� � �, particle filter computes the approximately

distributed of the N particles 1: 1 1: 1 1{ , }i i Nk k ix w� � �

according to the

posterior distribution 1: 1: 1:( , | )i ik k kp x w y .

B. SIFT Feature SIFT feature selects the keypoints of the image with

rotation invariance and stability against lighting or contrast changes, these keypoints are applied as descriptor for target objects in this paper. Difference of Gaussian (DoG) function is adopted to created SIFT feature as [18].

A keypoint descriptor is achieved by 2 steps as follows: 1) Gradient magnitude and orientation at each image pixel

around the keypoint location is calculated, as shown on the left side in Fig. 2.

2) Contents over 4x4 subregions are accumulated into orientation histograms as shown on the right side of Fig. 2, with the length of each arrow corresponding to the sum of the gradient magnitudes near that direction within the region.

Fig. 2 SIFT feature keypoint descriptor

To enhance the accuracy of tracking task, 4x4 descriptors computed from a 16x16 sample array is used as [18], and then the gradient orientation distribution is quantized into 128 bins.

Suppose the target object S has m SIFT keypoints, then

1{ }mi if ��s , (1)

where { , , , }p hif s o� is denoted as the SIFT feature vector of the keypoints, ( , )p x yp p� is the position of the keypoint of the image coordinate, s is the feature scale, o is the direction of the feature vector, and h is the 128 bins orientation histogram.

C. Garbor Feature In order to improve the discrimination capability of the

feature vector, Gabor functions are added to the feature space, as it is particularly suitable for image represent and exhibit strong characteristics of spatial locality, scale, and orientation selectivity. Gabor features of a given image can be created by convolving the Gabor kernels with the input image. The 2D wavelet kernel is product of a 2D Gaussian and a complex exponential function, and the resulting of Gabor function is expressed as follows:

2 2 2 2, ,

2( /2 ), /2

, 2( ) ( )u v u vk zu v ik zu v

kz e e e� ��

�� �� � , (2)

where u and v define the orientation and scale of the Gabor kernels, ( , )z x y� , and � denotes the norm operator, the wave vector ,u vk is defined as:

,ui

u v vk k e �� , (3)

where max /v vk k f� and / 8u u� �� . maxk is the maximum frequency and is usually set to 5, and vf is the spacing factor. In this paper, the parameters are selected as 8 orientation and 5 scales as [19], i.e. {0,1,...,7}u and {0,1,...4}v . The real part of Gabor kernels at eight orientations and five scales are shown in Fig. 3.

Fig. 3 Real part of Gabor kernels at eight orientations and five scales

Then the Gabor feature can be obtained by a 2D Gabor wavelet transformation of the image I as follows:

, ,( , ) ( , ) ( , )u v u vG x y x y I x y�� . (4) The dimensionality of the Gabor features of each pixel ( , )x y in the image I is 40. The Gabor feature can then be obtained as:

1 2 40[ , ,..., ]g g g�g , (5) where 8 ,u v u vg G� � � .

D. Region Covariance Matrices The SIFT features are invariabilities for illumination, scale

and affine, and Gabor features is particularly suitable for image represent and exhibit strong characteristics of spatial locality, scale, and orientation selectivity. To combine the advantage of the SIFT and Gabor feature, both of these feature are embedded into the RCMs, which presents several advantages as region descriptors and provides a natural way of fusing multiple features.

The RCMs is a matrix of covariance of several features obtained of a region in an image. Let I denote an image of size W H� , the d dimension feature vector iZ is defined by function as follows:

( , , )i I x y� Z , (6) where i y W H� � � is the index of pixel ( , )x y . Then the region can then be described by the d d� covariance matrix of feature iZ as

1

1 ( )( )1

nT

i ii

Cn �

� � �� � � u � u , (7)

where n is the total number of pixels in the region of the image and the mean vector u is obtained as

1

n

ii�

� �u Z . (8)

In the original version of RCMs [16], a covariance of several simple features is applied to representation the object, i.e. pixel position of image coordinate ( , )x y , color

( , , )R G BI I I�c and the first gradient magnitude xI and second order gradient magnitude xxI . Then the feature vector is extracted as

( , , ) [ ]x y xx yyI x y x y I I I I � c , (9)

where ( , )x

I x yIx

��

�and

2

2

( , )=xxI x yIx

��

.

The original RCMs is a way of fusing multiple features without normalization or weights distribution, which can take full advantage of these features in different spaces. Though these color feature and gradient magnitude features are relatively effective for tracking objects, their discriminating ability is not enough for the presence of background clutter, illumination varies and occlusion occurs. Inspired by the idea of RCMs, SIFT and Gabor feature are applied to represent objects to achieve well distinguished descriptor, the feature of the target object is extracted as follows:

[ , (1 ) ]� �� �F s g , (10) where 1{ }m

i if ��s , is the m SIFT keypoints in the object region as the Equation (1), g is the Garbor feature as the Equation (5), and � is a factor that balances the weight of SIFT and Gabor features.

In this paper, a novel particle filter based on SIFT-Gabor feature is proposed. The novel particle filter tracking weights each particle by consideration the feature of SG-RCMs. Then the approximately distributed of the N particles are computed according to the posterior distribution.

IV. EXPERIMENTAL RESULTS AND ANALYSIS

To evaluate the practical utility of the proposed SG-RCMs based objecting tracking, experimentally comparisons with traditional particle filter, SIFT based particle filter and Gabor based particle filter on a video is shown in this section. The experiments demonstrate that the proposed SG-RCMs based objecting tracking performance best, comparing with these other three methods.

The experimental result is shown as Fig. 4. The proposed SG-RCMs based objecting tracking well in the football tracking task. As the tradition particle filter is based on color histogram of interesting regions, the traditional particle filter miss the football to the socks, as the color distribution of the socks and the football is similar. SIFT based particle filter also fail to track the target for the SIFT points are scarcity. Although the window generated by Gabor based particle filter approximately catch the football, it is much large and does not give a precise result.

Fig. 4 Football tracking at sample frames using Traditional particle filter (blue), SIFT based particle filter (yellow), Gabor based particle filter (green), SG-RCMs particle filter (red).

V. CONCLUSION

In this paper, a novel SG-RCMs based particle filter is proposed. First, a new SIFT-Gabor feature is presented for object representation. In this step, RCMs is used for fusing SIFT and Gabor to take advantage of the two features. Then the particle filter framework is applied for object tracking. The experimental results show that the proposed method remains accurate and stable tracking performance, even though the object scale is changing over time or the background and the target has the similar color distribution, comparing with the traditional particle filter, SIFT based particle filter and Gabor based particle filter. Future work is to explore further ways for multi-object tracking.

REFERENCES

[1] H. Yang, L. Shao, F. Zheng, L. Wang, Z. Song, “Recent advances and trends in visual tracking: A review,” Neurocomputing, 74, pp. 3823-3831, 2011.

[2] A. Yilmaz, O. Javed, and M. Shah, “Object tracking: A survey,” ACM Computing Surveys, 38(4), 2006.

[3] T. Moeslund and E. Granum, A survey of computer vision-based human motion capture. Computer Vision Image Understand. 81(3) pp. 231-268, 2001.

[4] O. Javed and M. Shah, “Tracking and Object Classification for Automated Surveillance,” Lecture Notes in Computer Science, 2353, pp. 439-443, 2006.

[5] J. Courtney, “Automatic video indexing via object motion analysis, Pattern Recognition,” 30(4), pp. 607-625, 1997.

[6] C. Manresa, J. Varona, R. Mas, and F. Perales, “Hand Tracking and Gesture Recognition for Human-Computer Interaction,” Electronic Letters on Computer Vision and Image Analysis 5(3), pp. 96-104, 2005.

[7] R. Cucchiara, M. Piccardi, and P. Mello, “Image analysis and rule-based reasoning for a traffic monitoring system,” IEEE Transactions on Intelligent Transportation Systems, 1(2), pp. 119-130, 2000.

[8] C. Guo, S. Mita, D. McAllester, “A vision system for autonomous vehicle navigation in challenging traffic scenes using integrated cues,” IEEE Conference on Intelligent Transportation Systems, pp. 1697-1704, 2010.

[9] T. Broida and R. Chellappa, “Estimation of Object Motion Parameters from Noisy Images,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 8(1), pp. 90-99, 1986.

[10] R. Rosales and S. Sclaroff, “3D trajectory recovery for tracking multiple objects and trajectory guided recognition of actions.” IEEE Conference on Computer Vision and Pattern Recognition, pp. 117-123, 1999.

[11] H. Tanizaki, “Non-gaussian state-space modeling of nonstationary time series,” J. Amer. Statist. Assoc. 82, pp. 1032-1063, 1987.

[12] M. Isard and A. Blake. “Condensation–conditional density propagation for visual tracking.” Int. J. Computer Vision, 29(1), pp. 5-28, 1998.

[13] S. Arulampalam, S. Maskell, N. Gordon, and T. Clapp, “A tutorial on particle filters for on-line non-linear/non-Gaussian Bayesian tracking,” IEEE Transactions on Signal Processing, 50, pp. 174-188, Feb. 2002.

[14] P.Wu, L. Kong, F Zhao, and X. Li, “Particle filter tracking based on color and SIFT features,” IEEE International Conference on Audio, Language and Image Processing, pp. 932-937, 2008.

[15] H. Palaio, J. Batista, “A kernel particle filter multi-object tracking using gabor-based region covariance matrices,” IEEE International Conference on Imaging Processs, pp. 4085-4088, 2009.

[16] F. Porikli and O. Tuzel, “Fast construction of covariance matrices for arbitrary size image windows,” in Proc. IEEE Int. Image Process. Conf., pp. 1581-1584, 2006.

[17] F. Porikli, O. Tuzel, and P. Meer, “Covariance tracking using model update based on lie algebra,” in Proc. IEEE Comput. Vision Pattern Recog. Conf., pp. 728-735, 2006.

[18] D. Lowe, “Distinctive image features from scale-invariant keypoints,” International Journal of Computer Vision, 60(2), pp. 91-110, 2004.

[19] T. Lee, “Image representation using 2-D Gabor wavelets,” IEEE Trans. Pattern Anal. Mach. Intell., 18(10), pp. 959-971, 1996.