HAVE 2007 - IEEE International Workshop onHaptic Audio Visual Environments and their ApplicationsOttawa - Canada, 12-14 October 2007

Perceptual Surface Roughness Classification of3D Textures Using Support Vector Machines

Troy L. McDaniel', Sethuraman Panchanathan''Center for Cognitive Ubiquitous Computing

Department of Computer Science and EngineeringArizona State University, Tempe, Arizona, 85281

Abstract - Perceptual surface roughness classification describes medium and rough. (However, this approach is not limited tohow a surface's texture feels haptically in terms of perceptual these three perceptual categories.) Algorithms for perceptualcategories such as smooth, rough, bumpy, etc. Computer vision and surface roughness classification have a number of importantpattern recognition algorithms which estimate a surface's application areas includingperceptual roughness have a wide range of application areasincluding robotics, assistive devices, telesurgery and teleperception. 1) Robotics: humanoid robots must walk and runIn this paper, we propose a novel approach to perceptual surfaceroughness classification that, unlike previous approaches, is differently depending on many surfacedesigned to handle multiple roughness categories within the same characteristics, one being surface roughness.image. The steps of our approach include (1) texton extraction andclassification using a multi-class, non-linear Support Vector 2) Robot navigation: judgment of surface roughness,Machine; (2) segmentation using the Iterated Conditional Modes e.g., smooth, medium or rough, could invokealgorithm; and (3) overall perceptual roughness classification using different navigation plans and lead to bettera Nearest Neighbor classifier. The proposed approach is evaluated performance.using visio-haptic subjective measures of roughness on images ofthe 3D texture of real world objects. 3) Telesurgery: simple judgments of smooth or rough

.. ~~textures of internal organs can aid in importantKeywords - Image texture analysis, haptic user interfaces, visio- medical deiins..haptics.

I. INTRODUCTION 4) Remote teleperception of geographicalenvironments: perceptual judgments about surfaces

Humans have the uncanny ability to estimate how an can aid geologists and historians in makingobject feels in terms of its shape, size, texture, material, etc., important discoveries.entirely from its visual image [1]. From a biological Perceptual surface roughness classification is astandpoint, algorithms that estimate haptic (tangible) featuresfrom images mimic the human ability to transfer knowledge challenging research problem as algorithms must be robust tofrom one perceptual modality to another perceptual modality, (1) environmental variations including illumination changes,i.e. intermodaltrnfe.these algorithms ceabe considered scale changes, specularities, shadows, etc.; and (2) largei.e., intermodal transfer. Theserithm that can estimate haptic intra-class variations caused by the variety of different textureas visio-haptic transfer algorithms classes that fall into the same perceptual categories. This is ininformation from visual dlata.Pefrcetualf visiap .tan contrast to conventional texture recognition algorithms thatPerceptual visio-haptic transfer algorithms provide need only differentiate between images of different textureinformation about objects or surfaces at a perceptual level to nes.enable efficient user perception, rather than attempting We propose a novel approach to perceptual surfacereconstruct physical structures such as the 3D shape of an roughness classification that has been designed to be robustobject or the 3D texture of a surface. Teleperception systems to many environmental variations and large intra-classthat enable users to feel physically reconstructed surfaces or tony eover,nm cntrastnt and aroachssobjects from a distance can be augmented with perceptual variations. Moreover, In contrast to previous approaches [3]classification systems that communicate to users in terms of wesfirstpor segmationsbefore overall roughnessconcepts to achieve more efficient perception [2]. classficaton as images may consist of mulihple roughnessconcepts~~ ~ ~ ~ ~ ~ ~ ~~~ct onesTheev four mainensteepos oforarac.nldIn this paper, our focus will be on visio-haptic transfer ctgore Th orm tp oouapproc (nld(algorithms forperceptrfoua llsurfe rouness cl.asicatio. texton extraction; (2) texton classification using a multi-class,algorithms for perceptual surface roughness classification. .'

Perceptualsurface runscaIficatn. d.-e. o non-linear Support Vector Machine (SVM); (3) segmentation

surface's texture, as captured by an image, feels haptically in usn Itrae Codtoa Moe (1M 45;an 4vrl' ~~~~~roughness classification using Nearest Neighbor.terms of perceptual categories such as smooth, rough, bumpy, Semnaino ecpulruhesctgre a

rocky, slippery, etc. The proposed approach presented hereclsife a sufc'sruhesitooeo.he ayn provide useful information in many application domains. For

degres f sufac rouhnes, hichincudessmoth, example, in robot navigation, a mobile robot may want to

978-1-4244-1571-7/07/$25.OO ©2007 IEEE 154

avoid areas on the ground that are difficult to traverse. interreflections, specularities, etc., making 3D textureHowever, most approaches assume a single roughness recognition much more challenging compared to its 2Dcategory to be contained within an image, but this assumption counterpart [7].is not practical in real-world applications. Consequently, Approaches for 3D texture recognition can be broadlymuch information is lost when only the overall perceptual classified as (a) texton-based approaches; (b) color-basedsurface roughness of an image is considered, and often approaches; (c) Bayesian approaches; and (d) parametricincorrect results are usually obtained as algorithms are not models. Texton-based approaches [6,8,9] use filter banks ordesigned to handle such scenarios. directly use local pixel neighborhoods to extract textural

The rest of this paper is organized as follows. In Section 2, features called textons that can be used to recognize texturewe discuss background and related work. In Section 3, our classes, typically through the use of texton histograms andconceptual framework is presented. In Section 4, we present Nearest Neighbor classifiers. The use of textons for textureour experimental methodology. In Section 5, we provide a recognition is intuitive, and texton-based approaches havediscussion of our results. And finally, possible directions of provided impressive classification accuracy [9]. Color-basedfuture work are detailed in Section 6. approaches [10,11] exploit color information for texture

recognition. These approaches typically have shown highaccuracy on popular datasets, but have limited applicability.

II. BACKGROUND AND RELATED WORK Bayesian approaches [12,13] model the problem of 3Dtexture recognition in a Bayesian framework. Such

To date, only two approaches have been proposed for approaches have recently gathered much attention as aperceptual surface roughness classification: height maps and possible alternative to texton-based approaches to avoidspectral analysis, both proposed by Kahol [3]. We first lengthy training times. And finally, parametric modelsreview these approaches, and then we review approaches for [14,15] attempt to explicitly model a 3D texture's3D texture recognition as many of these techniques can be dependencies on illumination and viewing conditions, andutilized in the domain of surface roughness classification. can often be used for texture recognition and synthesis.

Kahol recently developed two novel techniques for However, parametric models in general are not expressiveperceptual surface roughness classification including height enough to effectively model a 3D texture's appearance undermaps and spectral analysis [3]. Our focus here will be on the varying lighting and viewing conditions, as manylatter as it has achieved higher classification accuracy of the assumptions about surface properties must be made. In thetwo approaches. First, illumination variation within training next section, the conceptual framework of our proposedimages is removed. Next, the 2D Fourier spectrum of each approach is presented.training image is found, and the middle 80% of coefficientsof the spectral transform are used to form 3D feature vectors.These extracted features are subsequently used to train a III. CONCEPTUAL FRAMEWORKPrincipal Component Analysis (PCA)-based classifier. Toclassify a novel image, it is first normalized for illumination, Our proposed conceptual framework, depicted in Figure 1,and its texture descriptors are computed using Fourier consists of four parts: (a) texton extraction; (b) textonanalysis. These descriptors are then classified using the PCA- classification; (c) segmentation; (d) overall classification.based classifier.

One limitation of this approach is that it is not designed to A. Texton extractionhandle multiple roughness categories within the same image.Although it is possible to divide an image into sub-regions The images used in this study consist of close-up, highand work on each individual sub-region, there are two resolution images of objects (everyday objects, handcraftedproblems with this approach: (1) determining the optimal size "nonsense" objects, etc.) captured under differentof each sub-region, and (2) algorithms that are designed to illumination conditions. (More details regarding the datasetwork at a global scale must now work locally with limited will be provided in the experimental methodology.) Theinformation, resulting in less accurate classification due to texture images of each object are grouped into roughnessnoise and limited data. Next, we review relevant approaches categories of smooth, medium or rough based on the resultsfor 3D texture recognition. of an experiment wherein blindfolded participants classified

The term 3D texture refers to real-world surfaces with the overall perceptual roughness of each object throughsurface height variations, as compared to 2D textures that touch. The details of this experiment, which is based on ourconsist of albedo or color variations. For 2D texture, the proposed methodology to collect reliable visio-haptic groundvariations in lighting and illumination are considered to be truth, can be found in [16]. This set of images will be referredglobal variations and can be easily accounted for through to as group]. Next, each image is manually segmented tobrightness scaling and affine transformations, respectively extract regions of smooth, medium and rough texture based[6]. On the other hand, 3D texture recognizers must account on the results of the aforementioned experiment. This groupfor complex changes in appearance due to changes in lighting of images will be referred to as group2.and viewing directions, which result in occlusions,

155

material class recognition [19]. Our approach to trainingSVMs for perceptual surface roughness classification issimilar to [18]; however, training is performed at a texton-level rather than at a texton histogram-level. Hence,perceptual surface roughness classification works locally and

Tegto Extraction can therefore handle multiple roughness categories within theandS Class~ication same image, which is commonly encountered in practice.

To train an SVM, the decision boundary, i.e., hyperplanew * x + b 0, between two classes of data points y7 =1+1,-i} for point xi is derived such that (1) all data points areclassified correctly, and (2) the decision boundary provides

Segmentation the maximum margin width. As (1) is usually not possible inpractice, i.e., data may not be separable by a linear decisionboundary, then a tradeoff, c, between the margin width and

_____ classification errors must be made. Specifically, we wish tofind the margin that minimizes

_~~~~~~~~~~~~~~~~ NOveraIIRoughness 2d NCas-icaon argminYwi +cE1 (1)

w,b i_, i_,

subject toy,(woxi+b)21- i, ei 20, Vi=I,.,N9Figure 1. Conceptual framework of proposed approach. First the

textons of an image are extracted and classified (blue, green and red where N is the number of data points and e. is the distance ofpixels indicate smooth, medium and rough textons, respectively). incorrectly classified point x to its correct location that

Next, textons are segmented. And lastly, the overall roughness of theimage is classified. would provide a correct classification. Eqn. (1) may be

solved through a Lagrangian formulation and quadraticEach image of group2 is subsequently converted to programming, not covered here.

grayscale and intensity normalized, which provides images As the SVM previously described does not handle non-with intensities of mean zero and unit standard deviation. To linear data, non-linearity may be introduced into theeach image, the Root Filter Set (RFS) [9] is applied. The RFS formulation through the application of kernel functions,consists of 38 filters with edge and bar filters at the 3 scales K(x,,x2) = cF(xl)e F(x2), where basis functions, ID(x),

(x )=(1,3), (2,6), (4,12)}, measured in units of pixels. map data points from their original space to a new, highFurther, each scale has 6 orientations. The RFS also includes dimensional space. Our proposed approach uses the Gaussiana low-pass Gaussian filter and a Laplacian of Gaussian (LoG) Radial-Basis Function (RBF) kernel function given itsfilter, both with G7=10. As the results of our experimental success in related literature [18], and is shown in (2).methodology will show, the dimensionality reductionMaximum Response 4 (MR4) [9] of RFS at a scale of (1,3) ( (x _x2 )2>provides the best classification accuracy, and hence it is the K(x,,x2)= exp 2 (2)filter bank we recommend for perceptual surface roughness K )classification. The MR4 filter bank allows invariance totexture rotations by taking the maximum filter response To handle more than two perceptual roughness categoriesacross all orientations of the oriented bar filters and across all (in this case three: smooth, medium and rough), a multi-classorientations of the oriented edge filters (at a specific scale). SVM may be used. Given its success [18], in this work weThis provides two filter responses, which, when taken use a tree-structured approach known as the Large Margintogether with the filter responses of the Gaussian and LoG Directed Acyclic Graph (DAG). At training time, an SVM isfilters, a 4-dimensional filter response vector, i.e., a texton, is learned for each pair of classes. During testing, only N-1obtained. After the filter bank has been applied to the entire SVMs need to be tested to determine the class of a data point,image, a set of textons is obtained, each labeled as smooth, making this approach very efficient.medium or rough from manual segmentation. The multi-class, non-linear SVM just described is trained

using the labeled textons of group2. However, as the numberB. Texton classification of textons per class will typically be very large, only a subset

should be used for training. We recommend randomlyFor texton classification, we propose the use of a multi- selecting M textons for each roughness category for use in

class, non-linear Support Vector Machine (SVM) [17]. SVMs training SVMs.have been very successful at 3D texture recognition [18] and

156

Classifying a texton as one of several roughness categories IV. EXPERIMENTAL METHODOLOGYis straightforward. First, an image is grayscale converted andintensity normalized. Next, the texton of each pixel is To evaluate our proposed approach, we used the 3Dextracted by applying MR4. Finally, each texton is texture images of the Visio-Haptic Object Database (VHOD)subsequently classified using the trained SVM. The [16]. The images consist of close-up, high-resolutionaforementioned classification steps, when applied to each 160x120 images taken of 48 different objects (everydaypixel in an image, typically results in noisy segmentation objects such as bowls, cups and glasses, and "nonsense"results. These results can be improved through the use of objects made from LEGO® building blocks). Each imageIterated Conditional Modes [4,5], described next. was manually cropped to remove background. For each

object, three images are taken under different illuminationC. Segmentation conditions created from two point light sources. Ground truth

in the form of perceptual surface roughness classificationsTo improve segmentation results, we can take into account was collected from blindfolded participants as they haptically

a texton's neighboring textons. We use the methodology explored each object to classify its overall roughness asproposed by [4], which uses the Iterated Conditional Modes smooth, medium or rough. The majority vote for each object(ICM) algorithm [5] to maximize a pixel's conditional was taken. We refer the reader to [16] for a detailedprobability based on its surrounding pixels. Each roughness description of ground truth collection.category is treated as an independent process, modeled by the Objects with significant visual textures, i.e., colorfirst order Gibbs-Markov random field: variations, were not used except for the handcrafted

"nonsense" objects built from LEGO® building blocks as_aNi color variations corresponded with surface height variations.

P(Ci N) =-e N (3) This left 18 smooth objects, 15 medium objects and 3 roughZ objects. Objects were randomly divided into training and

testing sets, providing 10 smooth objects for training and 8where Ci is class i, N is the size of the neighborhood, Ni is smooth objects for testing; 8 medium objects for training andthe number of textons within the neighborhood that belong to 7 medium objects for testing; and 2 rough objects for training

c is and 1 rough object for testing. These images make up imageclass i, Z iS the normalization factor (ignored here) and X S set groupi.the clique potential. The clique potential determines a To create image set group2, the training and testingtexton's dependence on its neighboring textons. As the images of group] were manually segmented to extract localabsolute value of A increases, this dependence strengthens. regions of smooth, medium and rough textural features.The posterior probability of (3) is used to classify each texton Segmentation of smooth textures involved extracting smoothof an image, i.e., the original class of a texton, as found regions that did not contain much specularity. Rough texturesthrough texton classification, may change if a different class did not require segmentation, and hence, images were usedproduces a larger probability in (3). Eqn. (3) is applied to directly for group2. Segmentation of medium textures waseach texton of an image until a stopping criterion is met, more challenging, but we relied on our observations duringtypically when the number of class re-assignments drops ground truth collection [16]. Typically, edges or areas ofbelow a threshold. edges or ridges in close proximity resulted in the perception

of medium texture by participants, and thus these featuresD. Overall classification were extracted as textural features of medium roughness.

SVM training and testing was performed on the images ofOften a user or system may require a summarization of group2. Five filter banks were tested including MR4 (3

perceptual roughness information, and hence a classification scales), MR8 and MRS4 [9]. For each filter bank, we testedrepresenting the overall perceptual surface roughness of an random sample selections of M=500, 1000 and 2000 samplesimage may be useful. To describe the roughness contained (beyond 2000 samples, training times were unreasonable).within an image, we propose a roughness descriptor of 3 Finally, for each filter bank and sampling combination, wefeatures consisting of the proportions of smooth, medium and tested different combinations of SVM parameters, namely 7,rough textons in an image. These descriptors, or training where y=0 5(o-2, specifically values 10 to 100 inmodels, are extracted from the training images of group] .

o

after texton extraction, texton classification and increments of 10, and c, specifically values 10 to 100 insegmentation, and are used in a Nearest Neighbor classifier. increments of 10 The best results were achieved with MR4 atGiven a roughness descriptor extracted from a novel texture, scale (1,3) with M=2000 samples, 7=20 and c=20. Thewe compute its Euclidean distance to each training model, classification accuracies for these parameter values are asand classify the roughness of the novel texture as the follows: 95.4% (smooth), 72.3% (medium), 77.8% (rough)roughness class of the model that is most similar to the test and 79.5% (overall). Using the aforementioned SVM thatsample in a Euclidean sense. achieved the highest classification accuracy determined

through preliminary testing, the textons of group] were

157

classified as smooth, medium or rough. Next, we describe the hence should be accounted for. Moreover, future work willuse ofICM to improve segmentation results. involve developing improved techniques for overall

Through preliminary testing, we found values for the roughness classification. Currently, the proposed methodparameters of (3), specifically, -0.1 and 7x7 for the clique does not take into account the spatial relationship ofpotential and neighborhood size, respectively. Moreover, we roughness categories. We believe that by exploiting thisfound a stopping criterion of 1000 class re-assignments to information, we can further improve overall roughnesswork well. Some segmentation results are shown in Figure 2. classification results.These results will be discussed in detail in the next section. Lastly, we wish to comment on the effect of visual

Finally, we evaluated our proposed approach for overall textures on our proposed approach. Visual textures willroughness classification. The segmented images of the obviously have an adverse effect on accuracy. However,training set, previously found, were used to train a Nearest preprocessing steps could be performed to remove visualNeighbor classifier. All training images were used except six textures such as the extraction of intrinsic images wherebyimages of smooth textures as these images had significant from a color image, a reflectance image and shading imageportions classified as medium or rough due to specularities can be extracted. Recently, significant progress has beenand noise. Classification accuracies of 91.7% (smooth), 100% made in this research area, and it would be interesting to test(medium), 100% (rough) and 95.8% (overall) were found. our approach on shading images of surfaces that haveThese results will be discussed in the following section. significant visual textures, and document the results. This

experiment will be part of future work as well.V. DISCUSSION OF RESULTS

REFERENCESResults have shown our approach to be robust to

illumination changes, specularities, shadows and noise. [1] Hatwell, Y., Streri, A. and Gentaz, E., Touching for knowing: cognitiveMoreover, as different textures were used during testing than psychology of haptic manual perception, John Benjamins Pub., Amsterdam,

thoseused uringtraiing, ur aproachis ale togenerlize Philadelphia, 2003.those used during training, our approach is able to generalize [2] Kahol, K., McDaniel, T. and Panchanathan, S., "Methodology forwell to novel textures, which is critical for many application efficient perception in exclusively haptic environments," In Proc. of IEEEdomains. As shown in Figure 2, segmentation results are International Workshop on Haptic Audio Visual Environments and theirpromising, and misclassifications during texton classification, Applications, pp. 140-145, 2006.

due to noise and specularities' arelargelycor[3] Kahol, K., "Distal Object Perception Through Haptic User Interfaces,"due to noise and specularities, are largely corrected through Ph.D: Arizona State University, 2006.the use of the ICM algorithm. Moreover, note that in Figure [4] Abdel-Hakim, A.E. and Farag, A.A., "Color segmentation using an2(c), edges are classified as medium texture, and in Figure Eigen color representation," In Proc. of 8th International Conference on2(f), transitions between smooth and rough textures, and vice Information Fusion, pp. 25-29, 2005.

versa, classified medium texture,which is what [5] Besag, J., "On the statistical analysis of dirty pictures," Journal of theversa, are classified as medium texture, which is what we Royal Statistical Society, 48(3):259-302, 1986.would expect and what was observed in [16]. [6] Cula, 0. G. and Dana, K. J., "Recognition methods for 3D textured

The overall classification accuracy for smooth textures is surfaces," In Proc. of SPIE Conference on Human Vision and Electronicsatisfactory at 91.7%. Misclassifications were a result of Imaging, pp 209-220, 2001.

specularities. Keep in mind that the majority of objects withn. [7] Dana, K. J., van Ginneken, B., Nayar, S. K. and Koenderink, J.J.,specularities. Keep in mindthat the majority Of objects within "Reflectance and Texture of Real-World Surfaces," In Proc. of IEEEour dataset have non-lambertian surfaces, and hence many Computer Society Conference on Computer Vision and Pattern Recognition,images have specularities. In any case, including both pp. 151,1997.lambertian and non-lambertian surfaces in the data set is [8] Leung, T. and Malik, J., "Recognizing Surfaces Using Three-

,oatvi tec r oter o apf Dimensional Textons," In Proc. of International Conference on Computerimportant to verify the accuracy of the proposed approach for Vision, pp. 1010-1017, 1999.use in real-world application domains. To improve overall [9] Varma, M. and Zisserman, A., "A Statistical Approach to Textureclassification accuracy, we can take into account the spatial Classification from Single Images," International Journal of Computerrelationship of textons, discussed in the next section. Vision, vol. 62, pp. 61-81, 2005.

[10] Kawewong, A. and Hasegawa, O., "3D texture classification by usingpre-testing stage and reliability table," In Proc. of IEEE International

VI. CONCLUSION AND FUTURE WORK Conference on Image Processing, pp. 11- 1330-1333, 2005.[11] Suen, P.-H. and Healey, G., "The analysis and reconstruction of real

We've proposed a novel approach to perceptual surface world textures in three dimensions," IEEE Transactions on Pattern Analysisand Machine Intelligence, vol. 22, pp. 491-503, 2000.

roughness classification that (1) is robust to many [12] Penirschke, A., Chantler, M. J., and Petrou, M., "Illuminant rotationenvironmental variations including illumination variations invariant classification of 3D surface textures using Lissajous's Ellipses," Inand noise; (2) can generalize well to novel texture classes; Proc. of the 2nd International Workshop on Texture Analysis and Synthesis,and (3) in contrast to past approaches, is capable of pp. 103-107, 2002.[13] Todorovic, S. and Ahuja, N., "3D texture classification using the beliefsegmenting an image into different roughness categories, net of a segmentation tree," In Proc. of International Conference on Patternwhich may be useful in a variety of application domains. Recognition, pp. 33-36, 2006.

Future work will investigate the effects of pose and scale [14] Dana, K. J. and Nayar, S., "Correlation model for 3D texture," In Proc.ofInternational Conference on Computer Vision, pp. 1061-1067, 1999.variations on the accuracy of our proposed approach. Pose [15] Dana, K. J. and Nayar, S., "Histogram model for 3d textures," In Proc.

and scale changes will undoubtedly be encountered in many of IEEE Conference on Computer Vision and Pattern Recognition, pp. 618-application areas such as robotics and teleperception, and 624, 1998.

158

[16] McDaniel, T.L., Kahol, K., Tripathi, P., Smith, D.P., Bratton, L., [18] Hayman, E., Caputo, B., Fritz, M. and Eklundh J.O., "On theAtreya, R., and Panchanathan, S., "A methodology to establish ground truth significance of real-world conditions for material classification," In Proc. offor computer vision algorithms to estimate haptic features from visual 8th European Conference on Computer Vision, 2004.images," In Proc. of IEEE International Workshop on Haptic Audio Visual [19] Caputo, B., Hayman, E., and Mallikarjuna, P., "Class-specific materialEnvironments and their Applications, pp. 94-99, 2005. categorization," In Proc. of IEEE International Conference on Computer[17] Vapnik, V., Statistical learning theory, Wiley, New York, NY, 1998. Vision, vol. 2, pp. 1597-1604, 2005.

(a)

(b)__

(c)

(d)**

(e)

(f)Figure 2. Segmentation results (blue, green and red pixels indicate smooth, medium and rough textons, respectively): (a)-(d) Testimages from the Vision-Haptic Object Database (VHOD) ((a) contains smooth texture; (b) contains medium texture; (c) contains

smooth and medium textures with an overall roughness of medium; (d) contains rough texture); (e)-(f) Images of novel textures, not inVHOD, taken with arbitrary capture conditions and parameters (in (e) both textures are rough, and in (f) the left and right textures are

rough and the middle texture is smooth).

159