Efficient and robust image descriptor for GUI object classification Anastasia Dubrovina, Pavel Kisilev, Daniel Freedman, Sagi Schein, Ruth Bergman HP Laboratories HPL-2011-200 Keyword(s): GUI object classification; image descriptor Abstract: Graphical User Interface (GUI) object classification is essential for image-based software automation tools.The challenges posed by GUI object classification are significantly different from those in natural imageclassification. In this paper we present a novel image descriptor developed specifically for GUI objects; itis robust to various changes in the appearance of GUI objects, such as various screen resolution, "skin", aswell as various operating system related. We use this image descriptor with Support Vector Machineclassifier, and experimentally show the descriptor robustness to the above transformations, and its superiorperformance compared to existing image descriptors.

External Posting Date: October 22, 2011 [Fulltext] Approved for External Publication Internal Posting Date: October 22, 2011 [Fulltext] Copyright 2011 Hewlett-Packard Development Company, L.P.

Efficient and robust image descriptor for GUIobject classification

Anastasia Dubrovina‡, Pavel Kisilev§, Daniel Freedman¶, andSagi Schein, Ruth Bergman‖

Abstract. Graphical User Interface (GUI) object classification is essen-tial for image-based software automation tools. The challenges posed byGUI object classification are significantly different from those in naturalimage classification. In this paper we present a novel image descriptordeveloped specifically for GUI objects; it is robust to various changes inthe appearance of GUI objects, such as various screen resolution, ”skin”,as well as various operating system related. We use this image descrip-tor with Support Vector Machine classifier, and experimentally show thedescriptor robustness to the above transformations, and its superior per-formance compared to existing image descriptors.

Keywords: GUI object classification, image descriptor.

1 Introduction

In this paper we address the problem of the Graphical User Interface (GUI)object classification. Examples of two types of GUI objects, namely push buttonsand edit boxes, are presented in Fig. 1. Other examples of the GUI objectsinclude radio buttons, check boxes, lists, menu item, etc. (see examples in Fig.4). The GUI object classification is essential for software automation tools basedon record/replay paradigm (such as [1]), where both the record and the replaystages are image-based, thus mimicking the human-computer interaction.

The main challenge in GUI object classification is the large variation in ob-ject’s appearance within a given class, even if the same operating system (OS) orbrowser are considered. For instance, the push buttons shown in Fig. 1 exhibit va-riety of sizes, colors, texts or images inside the objects. We assume that the GUIobjects differ by geometric and ”skin”-related transformations. The geometrictransformations may be of two types: (1) scaling (potentially anisotropic) dueto changes in screen resolutions, as well as naturally occurring differences such asedit box sizes, which can vary considerably in their aspect ratio; and (2) trans-lation. We assume that the GUI objects in question were found (or segmented)using some segmentation algorithm, and that the object may not be centeredwithin the subwindow provided by the segmentation. The ”skin” changes are

‡ Technion, Israel institute of Technology, nastyad@tx.technion.ac.il§ pavel.prvt@gmail.com¶ Bar Ilan University, Israel, d.e.freedman@gmail.com‖ HP Labs Israel, {sagi.schein,ruth.bergman}@hp.com

2 Anastasia Dubrovina et al.

(a)

(b)

Fig. 1. Snapshots of various object types: (a) push buttons and (b) edit boxes takenfrom applications running on Windows Vista operating system.

related to the variety of themes and colors provided by today’s operating sys-tems, and these changes are somewhat harder to quantify. Our approach is tocapture the look-and-feel information that remains relatively unchanged under”skin” transformations, and not color information that may change greatly.

In order to cope with the above challenges in GUI object classification task,we propose a novel type of image descriptor developed specifically for the GUIobject classification task. The proposed descriptor is based on the 1D versionof the Fourier-Mellin Transform. In addition, we incorporate information aboutobject image gradients and the percentage of the white color (we explain thedetails in the next section). We show experimentally that the proposed descriptoris robust to various geometric and skin-related transformations of GUI objects.Using the Support Vector Machine (SVM) classifier [9] along with the proposeddescriptor yields high correct classification rates for different GUI objects (seeSection 3).

Note that in this paper we do not use the context of the GUI object, such asthe surrounding objects, or the way the object changes its appearance in time, inorder to perform the classification. Such context information can be successfullyemployed together with the proposed image descriptor to boost the classificationaccuracy, as proposed in [5].

Efficient and robust image descriptor for GUI object classification 3

1.1 Previous work

In the context of image-based software automation products, such as ProjectSikuli [12], Prefab [4], EggPlant1, or related application for data masking -Magen[8], the object recognition is based on some variant of simple image tem-plate matching, or text matching, and therefore, cannot cope with the variabilityof intra-class transformations we discussed above. Project Sikuli [12] also em-ploys SIFT features [6], but these may not be informative for certain types ofGUI objects, as we show in Section 3.

Many state-of-the-art object classification and recognition methods use stan-dard features, such as SIFT, SURF [2], ASIFT [13], MSER [7], designed fornatural images. Further, as opposed to other object classification frameworks,for instance, PASCAL VOC Challenge 2, we deal with very small objects (sizeof a typical radio button is 12 × 12 pixels), of artificial nature. We have foundthat our image descriptor tends to work much better than the above featureswithin the framework of relatively simple objects such as screen-shots and GUIobjects.

The rest of the paper is organized as follows: we describe the details of theproposed image descriptor in Section 2. We provide the results of classificationof different GUI objects, experimental evaluation of the descriptor robustness,and compare it to the SIFT descriptor in Section 3. We conclude the paper withSection 4.

2 Image descriptor

Schematic description of the proposed GUI object classifier is presented in Figure2. For each GUI object we calculate the proposed image descriptor (described indetails below), using the grayscale image of the object. In order to perform theclassification, we first use a dataset of labeled GUI objects with their descrip-tors to train an SVM classifier, and then, given a new object, we calculate itsdescriptor and feed to the classifier to detect the object class.

In the following we describe the details of constructing the proposed imagedescriptor. It contains three different feature types.

1. Fourier-Mellin transform based feature. Given a grayscale image I, wefirst calculate the derivatives of its projections on x and y axes, as follows

Ix(x) =∂

∂x

(∑y

I(x, y)

), Iy(y) =

∂

∂y

(∑x

I(x, y)

). (1)

1 http://www.testplant.com/products/eggplant/2 http://pascallin.ecs.soton.ac.uk/challenges/VOC/

4 Anastasia Dubrovina et al.

Fig. 2. The proposed GUI object classifier scheme

Then we apply a 1D Fourier-Mellin transform to it. Schematically, it can berepresented as follows (shown here for Ix, but is the same for Iy):

Ix(xa − x0

) F−→ F{Ix}(a · u)eiux0|·|−→ |F{Ix}(a · u)| u→u=log u−−−−−−−→

|F{Ix}(u+ log a)| F−→ F {|F{Ix}|} (v)e−iv log a |·|−→ |F {|F{Ix}|} (v)|(2)

First, we take the Fourier Transform of the 1D signal, that is of the deriva-tives of the image projections. Then we take its absolute value; this operationeliminate the horizontal and the vertical shifts in the spatial domain. We thenuse only half of the spectrum, that is the positive frequencies, as it is sym-metric anyway. Next, we take the log scale in the frequency domain; thisoperation converts the scale into a new shift. Similarly, taking the secondFourier Transform and the absolute value eliminates the scale dependence.(Again, we use only positive frequencies). At the end, we get a shift and scaleinvariant representation, and it’s length is 1

4 of the length of the original sig-nal (since we halved the length by using twice only positive frequencies). Inorder to obtain a fixed-length representation for image of arbitrary size, wezero-pad Ix to have 1024 samples (or downsample it to have 1024 samples,if Ix is larger) before we calculate the first Fourier transform. Thus the re-sult of this 1D Fourier-Mellin transform is a vector of 256 samples. Afterwe concatenate transformed Ix and Iy signals, we have a 512-dimensionaldescriptor.

2. Histogram of gradient directions (angles) based feature We calculatethe angles of gradients in the grayscale image, wrap them to be in the range[0, π)), and calculate their 4 bin histogram. We weigh the gradient anglesby their magnitude, and also multiply them by triangle window weights

Efficient and robust image descriptor for GUI object classification 5

Fig. 3. An illustration of the descriptor (signature) computation.

according to how close they are to the bin centers, as proposed by DavidLowe in his article on SIFT. This descriptor helps mainly to separate radiallysymmetric objects such as radio buttons from the rest of the object classes.

3. Percentage of white pixels This descriptor calculates the percentage ofthe pixels with gray values larger than a predefined threshold (we used 245for this). This descriptor helps to separate edit boxes and push buttonsor other rectangular objects which are not empty (as edit boxes are). (Ingeneral, the above threshold can be learn from the database, though we didnot do it).

Concatenating the above three types of features results in a 518-dimensionalvector descriptor. Figure 3 illustrates the process of descriptor calculation. Notethat the proposed descriptor size is independent of the original object image size.

Finally, we would like to use the above image descriptor to perform the clas-sification of GUI objects using the SVM classifier. We use use the non-linearversion of SVM, namely Kernel SVM, where the kernel is taken to be the com-monly used Gaussian kernel between the descriptor vectors. That is, given twoimages of GUI objects, I1 and I2, and their descriptors f1 and f2, respectively,we measure the similarity between I1 and I2 by

K(f1, f2) = exp(−γ ‖f1 − f2‖22) , γ > 0. (3)

The Gaussian bandwidth γ is chosen by a cross-validation procedure.

3 Experimental results

In this section we show the results of the GUI object classification obtainedwith the proposed image descriptor. To perform the descriptor evaluation we

6 Anastasia Dubrovina et al.

(a) (b)

(c)

(d)

(e)

Fig. 4. Examples of different objects from our dataset: (a) radio buttons, (b) checkboxes, (c) list box openers, (d) edit boxes, and (e) push buttons ; all snapshots weretaken from applications running on Windows Vista operating system.

created a dataset of 258 segmented objects belonging to five GUI object classes:check box, edit box, list box opener, push button and radio button. Examples ofobjects from these five classes are shown in Fig. 4. We used this dataset both totrain the Kernel SVM classifier (details in the previous Section), and to performthe cross-validation procedure to find the optimal SVM parameters, as suggestedin [3]. In our code, we used the LibSVM Kernel SVM implementation [3].

3.1 Classification with the proposed image descriptor

In order to evaluate quality of the classification with the proposed descriptor werandomly divided the above dataset into non-overlapping training and testingsets. We used training sets of different sizes - 40% − 90% of the whole datasetsize. We performed 10 experiments for each one of these training set sizes - eachtime we randomly drew the necessary number of the training examples fromthe dataset. We then calculated the average number of the correctly classifiedobjects from all five classes, normalized by the test set size, as a function ofthe the training set size. The results are shown in Fig. 5, by the solid line withdiamond-shaped markers.

3.2 Descriptor persistence

In order to evaluate the robustness of the proposed descriptor to geometrictransformations we assembled a set of shifted, scaled and incorrectly segmented

Efficient and robust image descriptor for GUI object classification 7

Fig. 5. Percentage of correctly classified objects as a function of training set size,averaged over 10 experiments. The solid line shows the classification results for theproposed descriptor, and the dashed line shows the classification results for the BOWVcombined with the SIFT descriptor.

versions of an object from our dataset (see Fig. 6). We then trained the SVMclassifier using the objects from our dataset, excluding the transformed push but-tons, and classified transformed version of our object. All of them were correctlyclassified as push buttons. Note that since we are interested in GUI objects, theirsegmentation can be made relatively accurately, therefore examples of incorrectsegmentation in Fig. 6) may not happen in practice. Nevertheless, we saw thatwe can perform their classification even in those extreme cases as well.

3.3 Comparison with SIFT descriptors

In this section we compare the proposed descriptor to the state-of-the-art SIFTdescriptor [6]. In order to perform the classification we adopted the Bag Of (Vi-sual) Words (BOVW) approach used in [10]. As the GUI objects in question arevery small, we cannot rely on SIFT feature detector. Also, the high inter-classvariability in object appearance causes the SIFT feature detector find inconsis-tent features along the same class (say, text in push button - which is totallynot representative Fig. 7 shows an example of several push buttons with SIFTfeatures calculated using Vedaldi’s implementation of SIFT algorithm [11] forMatlab c©, with grayscale object images. No features at all were found for someof the smaller buttons.

8 Anastasia Dubrovina et al.

Fig. 6. Examples of possible transformations of a GUI object: shift, scaling, objectstate (before or after the mouse click), erroneous segmentation. The upper leftmostobject is the original push button from our dataset.

Fig. 7. Examples of push buttons with the detected SIFT features marked with bluecircles. The circle size represents the scale of the detected feature. Smaller objects haveno features detected.

Therefore, instead of searching for SIFT features, we used the image pixelswith high gradient magnitude value (higher than a predefined threshold of 10) asfeature points. For these feature points we calculated modified SIFT descriptors:we used fixed windows of size 8 × 8 around each feature point, and calculatedgradient histograms in its four subwindows of size 4× 4, thus obtaining descrip-tors of length 16. We then performed vector quantization of collection of thedescriptors obtained for all the objects, into 500 classes, and used the quantizedvectors as visual words. Finally, the object descriptors were calculated as his-tograms calculating the number of appearances of the above visual words in eachone of the object images.

We performed the classification as described in Section 3.1, using the aboveimage descriptors. The graph of the correct classification percentage is shown inFig. 5 above, dashed line, together with the correct classification percentage ob-tained using the proposed descriptors. Though the BOVW with SIFT descriptorsperforms quite well, the proposed descriptor yields better classification.

4 Conclusion

We presented a novel image descriptor designed for GUI object classification.We showed how it can be coupled with the Support Vector Machine classifier,to achieve robust classification of GUI objects from several selected GUI objectclasses, comparable to this of the existing image descriptors.

Efficient and robust image descriptor for GUI object classification 9

References

[1] O. Barkol, R. Bergman, D. Lehavi, A. Pnueli, S. Schein, A. Dubrovina, D. Freed-man, P. Kisilev, G. Noy, I. Marhaim, E. Cohen, Y. Livny, R. Nuriel, A. Nitsan,and D. Saaroni. Technology agnostic software testing. Submitted to TechCon2011.

[2] H. Bay, T. Tuytelaars, and L. Van Gool. SURF: Speeded up robust features.In Proceedings of the European Conference on Computer Vision (ECCV), pages404–417, 2006.

[3] C.-C. Chang and C.-J. Lin. LIBSVM: a library for support vector machines, 2001.Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm.

[4] Morgan Dixon and James Fogarty. Prefab: implementing advanced behaviorsusing pixel-based reverse engineering of interface structure. In Proceedings of the28th international conference on Human factors in computing systems, CHI ’10,pages 1525–1534. ACM, 2010.

[5] D. Lehavi et al. A New Parsing Language for GUI and Visually Structured Doc-uments. Submitted to TechCon 2011.

[6] D.G. Lowe. Distinctive image features from scale-invariant keypoints. Interna-tional Journal of Computer Vision, 60(2):91–110, 2004.

[7] J Matas, O Chum, M Urban, and T Pajdla. Robust wide-baseline stereo frommaximally stable extremal regions. Image and Vision Computing, 22(10):761–767, 2004.

[8] Sara Porat, Boaz Carmeli, Tamar Domany, Tal Drory, Ksenya Kveler, Alex Mela-ment, and Haim Nelken. Masking gateway for enterprises. In Orna Grumberg,Michael Kaminski, Shmuel Katz, and Shuly Wintner, editors, Languages: FromFormal to Natural, volume 5533 of Lecture Notes in Computer Science, pages177–191. Springer Berlin / Heidelberg, 2009.

[9] B. Scholkopf and A.J. Smola. Learning with Kernels: Support Vector Machines,Regularization, Optimization, and Beyond. The MIT Press, 2002.

[10] J. Sivic and A. Zisserman. Video Google: Efficient visual search of videos. InJ. Ponce, M. Hebert, C. Schmid, and A. Zisserman, editors, Toward Category-Level Object Recognition, volume 4170 of LNCS, pages 127–144. Springer, 2006.

[11] A. Vedaldi. An open implementation of the SIFT detector and descriptor. Tech-nical Report 070012, UCLA CSD, 2007.

[12] Tom Yeh, Tsung-Hsiang Chang, and Robert C. Miller. Sikuli: using gui screenshotsfor search and automation. In UIST’09, pages 183–192, 2009.

[13] Guoshen Yu and Jean-Michel Morel. ASIFT: An Algorithm for Fully Affine In-variant Comparison. Image Processing On Line, 2011.