Modeling context in haptic perception, rendering, and visualization

Modeling Context in Haptic Perception, Rendering,and Visualization

KANAV KAHOL, PRIYAMVADA TRIPATHI, TROY McDANIEL, LAURA BRATTON,

and SETHURAMAN PANCHANATHAN

Arizona State University

Haptic perception refers to the ability of human beings to perceive spatial properties through touch-based sensations. In haptics,

contextual clues about material,shape, size, texture, and weight configurations of an object are perceived by individuals leading

to recognition of the object and its spatial features. In this paper, we present strategies and algorithms to model context in haptic

applications that allow users to haptically explore objects in virtual reality/augmented reality environments. Initial results show

significant improvement in accuracy and efficiency of haptic perception in augmented reality environments when compared to

conventional approaches that do not model context in haptic rendering.

Categories and Subject Descriptors: H.5.2 [Information Interfaces and Presentation]: User Interfaces—Haptic I/O

General Terms: Human Factors, Design, Experimentation

Additional Key Words and Phrases: Haptics, haptic user interfaces, haptic cueing systems, context modeling

1. INTRODUCTION

Haptic user interfaces refers to systems and devices that enable touch-based interactions with digitalenvironments. (Haptics is a general term used to define the touch modality in the human sensorysystem [Hatwell et al. 2003].) Recent years have seen an increase in research and demand for hapticuser interfaces. There are various reasons for this increased level of interest.

With the growth of digital media and ready availability of memory and computing resources, there isan abundance of data. Computing environments are becoming extremely sophisticated and require highcognitive effort for the user to perceive and manipulate information. Research has shown that addinghaptic elements to the presentation of data can produce a manifold increase the usability of systems[Kahol et al. 2005a]. Hence, one of the major reasons for development of haptic interfaces is that thesedevices provide an exciting opportunity to develop multisensory feedback and communication systemsthat present data in an intuitive manner [Hale and Stanney 2004].

With the addition of haptic feedback and sensing units, there is a potential to increase veridicality ofthe virtual environments. Fields such as telesurgery and teleoperation are poised to gain substantiallythrough the development of haptic user interfaces. The demand for such systems has fueled researchin computational haptics.

Authors’ address: Center for Cognitive Computing, Arizona State University, Tempe, AZ 85287; email: {kanav,pia,troy.mcdaniel,

laura.bratton,panch}@asu.edu.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided

that copies are not made or distributed for profit or direct commercial advantage and that copies show this notice on the first

page or initial screen of a display along with the full citation. Copyrights for components of this work owned by others than ACM

must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, to redistribute to lists,

or to use any component of this work in other works requires prior specific permission and/or a fee. Permissions may be requested

from Publications Dept., ACM, Inc., 1515 Broadway, New York, NY 10036 USA, fax: +1 (212) 869-0481, or [email protected].

c© 2006 ACM 1551-6857/06/0800-0219 $5.00

ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 2, No. 3, August 2006, Pages 219–240.

220 • K. Kahol et al.

Another exciting possibility that arises from, research on haptic user interfaces is the developmentof universal access, assistive, and rehabilitative devices for individuals with sensory and cognitiveimpediments. In the past few years, multimodal systems and applications have proven to be an effectivetool in neural and physical rehabilitation [Tzovaras et al. 2004; Fritz and Barner 1997]. These systemsaid users in accomplishing tasks that target specific neural or physical impediments. Assistive andrehabilitative devices based on haptic user interfaces can address a variety of neural and sensorydisorders, such as vision impairment and memory impediments, with haptics playing a substantiverole in sensation, perception and cognition [Revesz 1950].

In spite of the considerable advantages offered by the development of haptic user interfaces, veridicalhaptic interfaces are restricted by the computational resources and tools [Salisbury et al. 2004]. One ofthe major limitations in haptic user interfaces is that they require a refresh rate of 1000 Hz for realisticfeedback. This lack of a high rate of feedback inhibits the development of interfaces that can allowmultipoint feedback simulations that mimic the haptic sensations experienced in real environments.Current haptic interfaces are limited to single point interaction systems that permit feeling virtualobjects one point at a time. This is an extremely inefficient method of interaction and works only in thecase of visio-haptic environments where haptics plays a supporting role. These systems are severelylimited for purely haptic environments or environments where haptics plays the role of major modalityof information exchange [Kahol et al. 2005b].

Another major obstacle to the development of haptic interfaces is the lack of psychologically andneurologically inspired haptic interfaces. The study of human tactile abilities is a recent endeavor andmany of the available systems still do not incorporate the domain knowledge of psychophysics, biome-chanics, and neurological elements of haptic perception. In this article, we present a psychologically andneurologically inspired haptic user interface for spatial object perception. The novelty of this system isthe incorporation of the psychological and neurological basis of haptics contextual models that influencethe haptic presentation of data. The following sections will show that research in the biological basisof haptics clearly highlights the role of context in haptic perception. In our system we model hapticcontext to:

(1) Develop a novel methodology to monitor hand movements employed to perceive objects, and predictthe haptic features the user is trying to perceive, and the perceptual classification of the objectfeatures.

(2) Develop a novel methodology for presentation of haptic data to the user. This methodology incor-porates the contextual model that a user employs to perceive objects in real environments. Simplecontextual cues invoke a haptic image or a concept of an object in the user’s memory. Resultsshow that this is an effective methodology for presentation of haptic data within a short period oftime.

2. THE ROLE OF CONTEXT IN HAPTIC PERCEPTION

Research on neurophysiology indicates that the hand has many types of haptic sensors that are special-ized to perceive different features of an object such as texture, shape, and material. Johansson [1996]conducted seminal research on the arrangement of tactile sensors in the human hand and showed thatdifferent regions of the hand are specialized to perceive particular spatial features. For example, theindex finger of the hand is specialized to perceive texture information while the tactile sensors in thepalm primarily perceive shape information. The physiological arrangement of tactile sensors and theirspatial arrangements allow humans to perceive various spatial features such as shape, texture, size,weight and material in parallel through multiple sensors in the hand, and to assemble a representationof objects and their features. However, static touch is not enough to build a memorial representation

ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 2, No. 3, August 2006.

Modeling Context in Haptic Perception, Rendering, and Visualization • 221

of objects. The exiguity of the tactile sensory system necessitates manual exploratory movements tohaptically perceive 3D environments [Hatwell et al. 2003].

Research in the psychology of haptic perception suggests that perception and action are closely re-lated in the haptic modality [Revesz 1950]. Various attempts have been made to study the manualexploratory procedures of individuals who are blind or sighted. A seminal study of exploratory proce-dures was reported by Lederman and Klatzky [1987]. They asked adults to use haptic exploration toclassify objects, according to a given criterion. This allowed them to identify specific exploratory handmovements (which they called exploratory procedures), which were characterized by

(1) the quantity and the nature of the information that each procedure provided, and

(2) the range of properties for which each procedure was useful.

Lederman and Klatzky reported that many of the exploratory procedures used by their participantswere related to the object property being explored, in a one-to-one relation. Some procedures are veryspecialized while others are more general. For example, lateral motion is meant only for perceivingtexture, unsupported holding for perception of weight, and pressure for perception of hardness of thematerial. Static contact principally gives information on temperature and more approximately on shape,size, texture, and hardness. Enclosure also gives global information on these properties, while contourfollowing provides precise knowledge about shape and size, and a vague idea about texture and hard-ness. Figure 1 shows some of the common exploratory procedures. Lederman and Klatzky also notedthat there are two distinct phases of the object exploration procedures in human adults. During the firstphase, they employ generalized procedures that mobilize the whole hand, and gather vague haptic andtactile information about several properties. During the second phase, specific exploratory proceduresare used to perceive particular object features. The regularity of hand movements utilized to perceivetactile features of objects is also depicted in the order of feature perception for objects with spatialregions larger than the grasp of the hand. Humans tend to perceive tactile features of large regionsin a serial order, systematically exploring object features piecewise, and assembling the instantaneousstimuli into a mental representation of the object. Nielsen [1995] studied this ability of humans toassemble serially perceived, instantaneous haptic stimuli into a unified tactile representation of anobject. The researcher specifically studied the effect of alteration of the haptic exploration process inchildren who are blind, to guide and teach them efficient strategies of exploration. Nielsen concludedthat any deviation from the natural strategy of the visually impaired child would decrease the percep-tual accuracy of tasks. She also noted that the only strategy for tactile search that is of value for thechild who is visually impaired is his/her own emphasizing the importance of the user-specific nature ofhaptic exploration. The user-specific nature of haptic exploration is manifested as stylistic differencesin individual haptic exploratory procedures as well as the sequence of haptic exploratory proceduresemployed. To a lesser degree, a user’s style is also manifested in differential sensitization of fingers indifferent individuals. For example, certain individuals use the thumb for fine texture perception ratherthan the index finger. This can be attributed to the mannerisms of the user.

Another more pronounced effect of the user’s context is the perception and memory aspect of haptics.Haptics, much like the other senses such as vision and audio, is guided by a user’s value system [Revesz1950]. For example, assigning perceptual categories to haptic object features such as texture, material,size, and shape is a user-centric activity. When modeling context, it is important to consider how anindividual perceives haptic sensations and how these percepts (discrete memory of the objects andsensations) can be invoked.

In summary, haptic perception by humans is a complex process that is influenced by threecontextual variables. Figure 2 summarizes the haptic perception process from the viewpoint of modelingcontext. The static grasp of the hand over an object and the placement of fingers on the object affect



Fig. 1. The common exploratory procedures. Copyright Lederman and Klatzky 1996.

Fig. 2. The role of context in haptic perception.



the information being perceived in parallel by different regions of the hand. The spatial position of thefingers and the palm could be modeled as spatial context in the haptic perception process. Spatial con-text allows dynamic adaptation of haptic stimuli based on the position of the palm and the fingers. Thisensures maximal perceptual accuracy by providing perceptually consistent stimulation in the hand.

The movement of the hands or exploratory procedures employed to perceive large spatial regionsand assemble sequential tactile stimuli into a complete object can be modeled as temporal context inthe haptic perception process. Research has shown the existence of regularity and structure in themovement employed to perceive haptic features such as texture, shape, size, etc. This allows modelingof the temporal context of haptic perception as a sequence of discrete hand movements: herein referredto as haptic gestures.

Last, we can model the user’s style of haptic perception and their mannerisms as user context. Thisallows customization of the haptic rendering and visualization schemes to the user’s style of hapticexploration and their cognitive strategy to assemble piecewise information into haptic object memory.

In this article, we present a methodology to model spatial, temporal, and user context in hapticperception with the aim of enhancing haptic visualizations, haptic rendering, and haptic user interfaces.There are three basic advantages to modeling context.

(1) Modeling context can enable the development of automatic systems that model haptic perceptionand haptic memory [Kahol et al. 2005b]. In real environments, humans use haptic experienceand memory to explore their surroundings. Contextual models allow developers to incorporateknowledge about haptic experience in the design of interaction paradigms. A case can be madefor contextual models of haptics by observing the development of visual displays and algorithms.Experiments in the psychophysics of vision led to the development of various standards such asPAL, NTSC, MPEG and JPEG, that model visual perception to provide efficient displays. We surmisethat modeling context in haptic user interfaces may lead to similar development of standards anddisplays.

(2) Context driven systems can lead to the development of adaptive displays that adapt to spatial,temporal, and user context in real time. Current haptic systems that render objects through basicalgorithms are limited by the resolution and feedback rates of haptic devices. Context driven systemscan enable better interaction between humans and computers by adapting displays to a user’ssensory, perceptual, and cognitive state.

(3) While haptic devices have been around for decades, not much is known about haptic perceptionin virtual environments. Contextual models will enable the development of better haptic user in-terfaces that allow for controlled psychological experiments that investigate haptic perception invirtual environments.

3. MODELING THE TEMPORAL CONTEXT AND THE USER CONTEXT

Psychological literature on exploratory procedures has restricted itself to the study of individual ex-ploratory procedures. However, global exploration strategy has not been studied in depth. One im-portant research question is: How do humans integrate various object properties perceived throughmanual exploration, namely shape, weight, texture, and size, to form a mental representation of anobject, and to recognize that object [Kahol et al. 2004]? In order to answer this question, it would behelpful to determine whether humans use a predictable strategy to explore objects, and whether itwould be possible to formalize the nature of that exploratory strategy. This strategy might be charac-terized in terms of the temporal order in which a human explores different types of object properties. Itis also important to determine whether there exists a common strategy for haptic object exploration, orwhether the strategies for exploration vary widely from one individual to another. Further, we address



Table I. User Information

Attribute Subject 1 Subject 2 Subject 3 Subject 4 Subject 5 Subject 6 Average and SD

age-group 18–29 30–45 18–29 30–45 45–60 18–29 32.4, 13.35

vision-loss age 14 0 0 23 45 0 —

visual ability light perception complete loss complete loss complete loss light perception complete loss —

the issue of the development of automatic systems that, based on manual hand movements, can predictthe feature the user is trying to perceive and the quantized value of the feature perceived. The objectiveof our research was to utilize hand movement analysis to achieve automatic recognition of exploratoryprocedures and thereby determine the feature the user is trying to perceive and the quantized valueof that feature. For example, we could analyze the hand movements to determine whether a user isperceiving texture, and to analyze the motion to further predict that the texture is perceived as rough.Such a system will allow ubiquitous and unobtrusive capture of haptic perception while a user exploresstimuli. This methodology would model the motor strategy of exploration and further model the user’svalue system that attributes perceptual categories to the perceived stimuli.

To achieve our goals, three experiments were designed. The first experiment was designed to de-termine the exploration strategy of individuals when perceiving objects. The second experiment wasdesigned to develop hand movement recognition engines. The third experiment was designed to predictthe perceived value of features based on hand movement analysis. Together the cognitive strategy andthe motor strategy represent a formal model of the temporal context, and partially the user context asdefined in our approach.

3.1 General Experimental Procedure

The experiments involved six participants who were legally blind. Table I gives information aboutthe participants. We chose to work with individuals who are blind since the long term goal of ourresearch is to develop assistive devices that enable the perception of distal objects for individuals whoare blind. A pre-experiment questionnaire was given to each participant. This questionnaire was usedto gather data about the participants’ state of vision loss, their age, and the age at which they losttheir vision. In order to ensure consistency among participants and to eliminate any bias due to partialvisual perception, those individuals who were not totally blind were made to wear blindfolds. The datacapture setup consisted of three digital video cameras, a pair of CyberTouch data gloves, and Ascensiontrackers. The gloves captured the angle of each joint in the hand, and the trackers captured the (x,y, z) position as well as the 3D orientation of each hand in space. The gloves are one-size-fits-all withholes in the fingertips, enabling the fingertips of the hand to make direct contact with objects. Thevibro-tactile motors located at the back of each finger are lightweight and unobtrusive. The tracker,located at the back of the wrist on each hand, is lightweight and unobtrusive. When asked about thegloves, all participants reported that the gloves were comfortable and did not obstruct how they wouldnormally perceive an object haptically. Data streams were captured from the trackers and from the twodata gloves (right and left hand) at 120 frames/sec. In all experiments, the participants wore the gloves(augmented with the trackers) and were seated in front of a table on a chair with no arm support. Theparticipants were then asked to hold the palms of their hands facing upwards, and an object was placedon their hands. The participants then explored the objects with their hands in their natural manner,while data streams were captured from the gloves and the trackers. In addition to recording these datastreams, the experimental apparatus also recorded the exploration process in real time with the threedigital video cameras. The first step of the experimental procedure was to conduct a training phase.The objective of this phase was to familiarize the participants with the gloves and the verbalizationprocess. The instructions were read aloud to each participant from a written script, which ensured



Fig. 3. Objects used in the experiment. Copyright Kahol et al. 2004.

that the instructions were consistent across participants. During this phase, participants were giventwo test objects: a small teapot and a cup. The identity of the objects was revealed before handingthe object to the participant, and the participant was told to explore the objects and confirm the givenidentity. While exploring, the participants were asked to verbalize their exploration by verbally statingthe characteristic and/or the feature they were exploring. This exploration was done with the elbowselevated above the surface of the table, as this allowed capture of the unperturbed hand movements.This training phase was interactive, with the investigators’ general feedback on experiment-relatedquestions.

3.2 Experiment 1: Modeling the Cognitive Strategy of Haptic Exploration

Experiment 1 consisted of two phases, called the free-running phase and memory phase. During thefree-running phase, participants explored the objects as they verbalized their strategy of exploration.This verbalization was captured with the three digital video cameras, and by an observer who tooknotes during the capture process. Five objects were presented to each participant during this phase.These objects were randomly chosen from a set of 12 objects, which are shown in Figure 3. One ofthe goals was to compare the exploration strategy used during the free object exploration phase to thetask-based exploration strategy used during the memory phase to determine the degree to which theexploration strategies of each participant were consistent across both situations. The memory phaseemployed task-based exploration that required users to compare pairs of objects, and then indicatethe similarities and dissimilarities between those objects. Six pairs of objects were presented to eachparticipant during the memory phase. Each pair involved a controlled variation of one parameter. Theparameters chosen were: shape, texture/material, size, weight, state of being filled or empty, and stateof being open or closed. Figure 4 depicts the objects presented for the memory phase.

3.2.1 Data Analysis. The video data gathered from the free-running phase and the memory phasewas subsequently annotated using Anvil video annotation software [Kipp 2001]. (Different annotationschemes were used for the two phases.) The start and end frame numbers were marked for the explo-ration of each object feature. Then, for each object, the order of exploration was noted, in terms of shape,



Fig. 4. Objects used in the memory phase. Copyright Kahol et al. 2004.

weight, texture, identification, and object-specific exploration patterns. For example if a participant firstexplored shape, then texture, and then weight, the annotation for that participant was shape, texture,weight. For each object feature, the exact amount of time spent on exploration was noted. Using theprocedure given above, videos for each of the tasks and each of the six participants were annotated. Foreach participant, the order of exploration of the object features was recorded for each object, and themost consistent order was determined. A standard deviation was calculated between objects’ order ofexploration employed for various objects.

3.3 Experiment 2: Computational Modeling of Hand Movements of Haptic Exploratory Procedures

In this experiment, each participant was given five objects. For each object, the participant was askedto perform ten tasks. These tasks included perceiving

(1) the texture of the object and classifying it as rough, medium, or smooth,

(2) the size of the object and classifying it as big, small or medium,

(3) the curvature of the object and classifying it as no curvature, small curvature, medium curvatureor high curvature,

(4) the rim of the object and classifying the shape as round, square or irregular,

(5) whether the object was open or closed,

(6) whether the object was filled or empty,

(7) the edges of the object,

(8) the base of the object and classifying it as round, square or irregular, and

(9) classifying the object.



(The information on classification of features into perceptual categories, such as big, small, or medium,was used for experiment 3.) For each of the six participants, the objects were selected randomly fromthe set of 12 objects shown in Figure 3, with no repetitions. To analyze the exploratory hand movementsused to perceive the object features, we selected the set of hand movements for each of the featuresmentioned above.

3.4 Data Analysis

Since there were 6 participants, and 5 objects per participant, Experiment 2 yielded 30 explorationsequences for each exploratory procedure. In order to develop a personal hand movement vocabulary,we trained and tested Hidden Markov Models (HMMs) for each set and for each user. The followingsection defines the mathematical foundations of Hidden Markov Models.

3.4.1 Hidden Markov Models. HMMs are probabilistic modeling tools employed for temporal se-quence analysis, and have been widely used in gesture and speech recognition [Rabiner and Juang1986]. An HMM models a temporal sequence of events (called an observation sequence) in terms of astate machine, in which the current state of the model is probabilistically dependent on the previousstates. The HMMs have a finite number of states. Transitions between states are governed by transitionprobabilities. In every state, an output symbol can be generated according to an associated probabilitydistribution. Further, there is a probabilistic function that governs the beginning state in the statemodel. This is called the initial probability distribution. Only the observation symbols are available toan external observer; the state generating the symbol is hidden.

Mathematically an HMM λ can be defined as

λ = (A, B, π ) (1)

where A refers to a set of transition probabilities such that A = {a(i, j )}, where a(i, j ) represents transitionfrom state i to state j .

a(i, j ) = p{qt+1 = j |qt = i}, 1 ≤ i, j ≤ N (2)

where N is the number of states and qt denotes the current state.Transition probabilities satisfy the following constraints

a(i, j ) ≥ 0,N∑

j=1

a(i, j ) = 1, 1 ≤ i ≤ N . (3)

B is a set of probability distributions in each of the states {bj (k)} such that bj (k) represents the proba-bility of generation of observation symbol k at state j .

bj (k) = p{ot = vk|qt = j }, 1 ≤ j ≤ N , 1 ≤ k ≤ M (4)

where vk denotes the kth observation symbol in the alphabet, ot is the current observation symbol,and M is the number of alphabets or observation symbols. The following stochastic constraints must besatisfied, where π is the initial state distribution.

M∑k=1

bj (k) = 1 (5)

π = p{q1 = i}, 1 ≤ i ≤ N . (6)

There are three basic problems of defining and using an HMM.



Fig. 5. Sample Hidden Markov Model. s1 represents sample 1 and s2 represents sample 2.

The Learning Problem pertains to adjusting the parameters of an HMM λ in order to maximizethe probability of production of a given sequence O by a given HMM λ, which is denoted by p(O|λ).Two types of optimization criterion are used to solve the learning problem. The Maximum Likelihoodcriterion is employed to maximize the likelihood of generation of a sequence from an HMM by adjustingthe parameters. The Baum Welch algorithm, and gradient based methods are common algorithms thatuse this criterion for learning. The Maximum Mutual Information criterion amounts to training theentire HMM library consecutively. This criterion maximizes the probability of production of a trainingsample from the correct class while minimizing its production probability from other HMMs. Details ofthese algorithms can be found in Rabiner and Juang [1986].

The Decoding Problem helps identify the state sequence in an HMM that has the highest probabilityof generating a sequence of observations O.

The Evaluation Problem helps determine the probability of a particular observation sequence beinggenerated by an HMM λ, p(O|λ).

See Figure 5 for an example of an HMM. In this case, the objective is to train an HMM to model agesture as shown in Figure 5. In this case, we choose to model the gesture as a sequence of 3 poses.These three poses represent three states of the HMM. The set of probabilities {ai, j} represents the setof transition probabilities between the three states. This is practically achieved by time normalizinga sequence, dividing it into 3 equal sections, and then choosing the beginning pose (or end pose) ofeach region as being a state. In real environments, it is not always possible to repeat three posesexactly in a similar manner. Consider samples of the same gesture. These samples can vary in speedand other parameters, but are still examples of the same class. In the case of HMM training, at everystate it is hence possible to have different variations of a pose. These variations form the alphabets orobservation symbols. In Figure 5, we consider 2 samples of a gesture. As there are 2 samples and 3states, there will be possibly 6 observation symbols, as shown in the figure. It may be noted here that



when multiple samples for training are involved, many poses may map onto a single observation symbolthereby reducing the number of symbols in the formulation. However, in this case, we consider all sixposes to be separate observation symbols. The set of probabilities {bj (k)} represents the probabilityof production of a pose in a given state. During learning, the HMM parameters A, B, � are tuned tomaximize the probability of sample 1 and sample 2 belonging to the gesture class represented by theHMM. Given a sample gesture and a trained HMM we can determine the probability of it belonging toa particular gesture class. The algorithm for the decoding problem can be implemented to determinethis quantity.

Various extensions have been proposed to the standard HMM model. One such extension involvesmultiple HMMs coupled together to produce certain sequences by having transition probabilities be-tween the states of the involved HMMs. For C chains coupled together, the state transition probabilityis

p(

S(c)t

∣∣S(1)t−1, S(2)

t−1, S(3)t−1......S

(C)t−1

), 1 ≤ c ≤ C (7)

instead of the p(S(c)t |S(c)

t−1) in the standard HMM. Zhong and Ghosh [2002] defined algorithms fordistance-coupled HMMs in which the transition probability from a state of one HMM to a state inanother HMM depended on a variable they called distance between the two nodes. In their method,the standard formulation for HMM is extended to include a set of coupling coefficients � that definesdistance between the nodes of coupled HMMs.

3.4.2 Modeling Exploratory Procedure as a Sequence of Poses. In our first formulation, an ex-ploratory procedure is modeled as a sequence of poses, each of which is represented by a state ofthe HMM. The following is the methodology. Every gesture represents a different class and is modeledby a separate HMM. As we had 9 types of exploratory gestures, we developed a library of 9 HMMs:one for each class of gesture. For every gesture, HMMs were trained as follows. Captured sequencesof a class of gestures are time-normalized to a standard time duration. This is achieved by reducingthe duration of gestures to the example with the minimum duration. Time normalized sequences aresequences of frames wherein each frame consists of coordinates(< x, y , z >), values of hand joints, aswell as angles at joints. We chose to represent each gesture using 4 states. This is achieved throughdividing the time-normalized sequence into 4 equal parts and choosing the frame at the end of eachsegment as an observation symbol at that state. An HMM representing a gesture is trained using theseobservation symbols through the learning algorithm. This methodology is repeated for each gestureand gives the trained HMM library. In order to determine the class of a given gesture during testing,the given sample is divided into 4 equal parts (as we had 4 states) and the frame at the end of eachpart forms the observation symbol at each of the states. This sequence of poses is passed through eachof the 9 HMMs, and the probability of generation of the test sequence from each HMM is calculatedusing the algorithm for the evaluation problem. The HMM that generates the maximum probability forthe given test gesture is identified as the class to which a test gesture belongs. Recognition accuracy ofthe HMM library is measured as the number of correctly recognized gestures as compared to the totalnumber of test gestures.

4. EXPERIMENT 3: ESTIMATING SPATIAL FEATURES THROUGH HAND MOVEMENT ANALYSIS

Experiment 3 was designed to find computational models that can predict perceived values of featuresfrom hand movement analysis. In Experiment 2, we gathered data on hand movements over objectsas well as the perceived value for the feature. We selected a subset of features and its classes forExperiment 3 as depicted in Table II. In the previous section, we discussed pose based modeling ofexploratory procedures. While this modeling is ideal for simple movement vocabularies, it does not



Fig. 6. Kinematic model of the hand.

provide an adequate basis to distinguish between movement patterns for different perceptual classesof various features. An alternative modeling of exploratory procedures was needed for the purposes ofExperiment 3. The key to identifying an alternative modeling of gestures is to recognize that all of themotion sequences that can be performed by the human hands are constrained by the anatomy of thehand. At a mechanical level, the human hand can be modeled as a complex system of hierarchicallyconnected rigid units as in Figure 6. This kinematic model has 27 degrees of freedom (DoF). Each ofthe four fingers has four DoF. The distal interphalangeal (DIP) joint and proximal interphalangeal(PIP) joint each have one DoF, and the metacarpophalangeal (MCP) joint has two DoF due to flexionand abduction. The thumb has a different structure from the other four fingers: it has five degreesof freedom, one for the interphalangeal (IP) joint, and two each for the thumb MCP joint and thetrapeziometacarpal (TM) joint, both due to flexion and abduction. The fingers together have 21 DoF.The remaining 6 degrees of freedom are from the rotational and translational motion of the palm with3 DoF each.

This structure of the joints and the hand segments remains constant in able bodied individualsand suggests that a modeling of gestures based on events occurring in these segments and jointswould provide an adequate basis for coding and recognizing hand gestures. In the proposed approach,events in the motion trajectory of each of the segments are detected as local minima in the segmentalacceleration. Events in joints are detected as stabilization of the angles between adjacent segments atthe joints. Stabilization of the angle at a joint corresponds to a period during which the change in theangle does not vary beyond a threshold range. It was observed that at a sampling rate of 80 frames/sec,only 1 segment trajectory passes through a local minimum and only one joint exhibits stabilization,thus making this model physiologically plausible.

In order to test this modeling, we propose a human-anatomy-based event-driven HMM approach.The proposed approach employs two HMMs. The first HMM has 15 states—each state representing theevents occurring in a particular hand segmentHMM. This HMM is called the segmentHMM. The modelenters a given state of the segmentHMM when the segmental force associated with that state arrivesat a local minimum and outputs the force value at that moment. The second HMM has 23 states—eachstate representing the events occurring in a particular joint in the hand. This HMM is referred to as thejointHMM. The model enters a given state of the jointHMM when the angle at the joint associated withthat state has stabilized (i.e., the angle variation over time has been below a threshold for a specifiedtime interval), and outputs the angle at the joint. In order to represent hand gestures, these two HMMsinteract to produce a single coupled HMM (cHMM). Each state of the segmentHMM is coupled with allof the states in the jointHMM. The states of the two HMMs are coupled based on distances between



Table II. Features and Corresponding

Classes Used for Experiment 3

Feature Perceived Classes

Texture smooth, medium, rough

Curvature small, medium, high

Rim Shape circular, square, irregular

Base Shape circular, square, irregular

Size small, medium, big

segments and a joint. This body distance coupling is derived from the work of Zhong and Ghosh [2002].The joint transition probability of the coupled-HMM is given by

p(

S(c)t

∣∣SsegmentHMMt−1 , SjointHMM

t−1

)=

2∑c′=1

�c,c′ p(

Sct

∣∣Sc′t−1

). (8)

In our case, �c,c′ , is function of the body distance between segments and joints. It is defined for everysegment as the sum of the inverse of the body distances between all of the joints and the given segmentfor coefficients from jointHMM to segmentHMM.

�c,c′ =(

14∑t=1

1

d (t, i)

)(9)

for state i of segmentHMM.

�c,c′ =(

23∑t=1

1

d (t, i)

)(10)

for state i of jointHMM. Zhong and Ghosh [2002] described an extended learning algorithm, and anextended forward backward procedure that were used in our work for training and testing purposes.We defined 12 different HMMs for each of the perceptual classes defined in Table II for every user. Asis clear, not all of the categories captured in Experiment 2 were considered. This was due to data notbeing sufficient in some categories such as no curvature. For each class, 15 captured hand movementswere used for training, and 15 captured hand movements were used for testing. Recognition accuracyfor each of the testing gestures was noted.

4.1 Results

Table III shows the most consistent order of exploration for each participant, and the standard deviationfrom that order, which was noted in Experiment 1. Table IV shows the recognition accuracies for eachof the exploratory procedures in Experiment 2. Table V shows the results of Experiment 3.

4.2 Discussion of Results

The cognitive strategy can be summarized in terms of 3 processes: (1) Object class identification (2)Object-specific exploration and (3) Object characterization. Object class identification is the act of rec-ognizing what class of object it is. (The object class was identified almost instantaneously by all ofthe six participants.) Following class identification, participants tended to look for features of theobject that distinguished it within that class, a process that we called object-specific exploration.Following object identification, participants tend to confirm their hypothesis about the object class.It is during the object-specific exploration that the exploration strategy starts to show signs of individ-uality. It is significant to note here that the temporal order of exploration for each participant was the



Table III. Results of Cognitive Strategy. R,C: Rim and Curved Regions, I: Inside, B: Base. Texture Here Refers to

Surface Properties in General

Subject 1 Subject 2 Subject 3 Subject 4 Subject 5 Subject 6

object recognition object recognition object recognition object recognition object recognition object recognition

object-specific

exploration

object-specific

exploration

object-specific

exploration

object-specific

exploration

object-specific

exploration

object-specific

exploration

texture shape (R,C) shape (R,C) texture size size

shape (R,C) texture shape (B) texture texture weight

shape (B) texture shape,texture(I) shape (R,C) shape (R,C) shape,texture(I)

size shape (B) texture shape (B) shape (B) texture

weight weight texture shape,texture(I) texture shape (R,C)

texture size weight weight shape,texture(I) shape (B)

shape,texture(I) shape,texture(I) size size weight texture

standard deviation

0.2

standard deviation

0

standard deviation

0

standard deviation

0

standard deviation

0.2

standard deviation

0

Table IV. Recognition Accuracies for

Exploratory Procedures. Copyright Kahol et. al.

2005aSpatial Feature or Task Recognition Percentage

texture/material 99.00

size 100.00

curvature 97.80

rim 100.00

open or closed 100.00

filled or empty 100.00

edges 99.00

base 100.00

recognition 100.00

Table V. Recognition Accuracies for Perceptual Classification

Feature and Value Recognition Percentage

texture: rough, medium, smooth 100.00, 100.00, 100.00

size: big, small, medium 100.00, 100.00, 100.00

curvature: small, medium, high 100.00, 100.00, 100.00

rim shape: circular, square, irregular 100.00, 86.6, 86.6

base shape: circular, square, irregular 100.00, 100.00, 100.00

same during free exploration of the objects and during the comparison of pairs of objects in the memoryphase.

The results of these Experiments suggest that the plan used for haptic exploration is consistentacross various tasks. The very high accuracy in the prediction of object feature classes based on thevocabularies of exploratory hand movements suggests that humans use personalized hand movementsto perceive certain types of feature. These results suggest a common methodology to model temporaland user context. The methodology requires a training phase, wherein users are requested to explore acertain number of objects and their features, while their hand movements are captured and analyzed.The hand movements are used to train a library of Hidden Markov Models that are subsequently usedto model motor strategy and the perceived features. The cognitive strategy is modeled as the consistentobject exploration strategy as recorded from the training phase. Real time hand movement analysiscould reveal what feature a user is trying to perceive, and the display could be adapted to present theinformation in a focused manner.



5. TACTILE CUEING AS A VALIDATION SCHEME FOR CONTEXTUAL MODELS

Section 4 described the psychophysical experiments designed to capture the user’s style of explorationas well as the temporal context models. In this section, we first describe the design of an experimentalparadigm that allows us to test spatial, temporal, and user contextual models.

Current haptic interfaces such as the CyberTouch gloves, place haptic feedback units on differentregions of the human hand. These devices could be programmed to communicate information in parallelto the user, and cue a user about certain information relevant to the virtual environment. Hapticcueing systems present an alternative to conventional haptic rendering of visual forms [Kahol et al.2005b]. It is analogous to the audio-visual messaging used in conventional GUIs, where the user’sattention is attracted to an event through audio-visual cues. For the purposes of testing our contextualmodels, we propose the development of haptic cueing systems that can convey information about thevirtual environment by supporting or replacing conventional haptic rendering. Multimodal interfaceswith haptic cueing have been used in various applications such as generation of alerts in cars forpossible collisions [Tan et al. 2003]. However, conventional cueing paradigms are limited to focusingthe attention of the user, informing him or her of the event occurrence through tactile feedback. Also,currently available cueing systems were developed for multimodal environments, which include visualfeedback, and hence are not suited for haptic systems for individuals who are blind. The key conceptualframework that guides this approach is that humans have haptic memory of an object and, with sparsedata about the object features, can perceive and guess the identity of the object. For example, considerthe following statement, and visualize an object about which this statement is made: The object is a glassmade of plastic, is of medium size, and its texture is smooth. Moreover the object’s base is smaller thanits rim and its vertical surface from rim to the base has no curvature. A majority of readers would concurwith the view that this statement if accurate, provides enough information to invoke a mental imageof the object. This mental image is not necessarily specific to a single modality but rather involvesmultiple modalities. Our proposed system intends to utilize this human ability to recognize objectsthrough simple cues about its features.

We designed a series of experiments that involved 5 participants who were blind (subject 6 from theprevious experiments could not participate in this set of experiments due to personal reasons).

The training involved the subject exploring a set of 5 objects and quantifying their texture, shape, size,and material into predetermined quantization levels for each feature. Then the participant determineda tactile code to communicate the quantization level for a particular feature. The code was designedto be a combination of three pulses that are tactile analogies of “dot” or “dash” visual patterns inMorse code. For example, the participant chose “dot,” “dot,” “dot” to represent small size, “dot,” “dash,”“dot” to represent medium size and “dash,” “dash,” “dash” to represent large size. A lexicon of codesrepresenting feature, quantization level, and code, was created for each of the participants. In order toensure consistent training, each user began with the same lexicon. A software was designed to read inthe lexicon for a participant, and then based on the object feature values fed in, automatically send theuser the tactile cues. The software allows users to change their lexicon and add new codes to the lexicon.

Each of the vibrotactile motors is dedicated to presentation of a particular feature. The assignment ofa feature to a finger motor was based on the neurophysiological distribution of tactile sensors (spatialcontext as defined in Section 2). For example, texture information was assigned to the index fingervibrotactile motor as the index finger contains Merkel receptors [Johansson 1996] that are specializedto perceive texture information. In a similar manner, shape information is sent to the palm, materialinformation is sent to the ring finger, size information is sent to the thumb, and so on. This techniqueencoded the spatial context in our approach.

A training phase was employed for each of the users. In the training phase, users were presented withhaptic cues and were requested to recall the quantization level represented by the code. To present codes



Fig. 7. Division of object for cueing.

for the entire object, each object was divided into 7 regions. These regions were numbered based on theorder in which the cues would be presented. Figure 7 shows the division of the object into the 7 regions.The first phase of cues provided information about the overall shape, size, texture, and material of theobject (region #1: overall object). The second phase of cues was programmed to convey information aboutthe object’s base in terms of its shape, size, texture, and material (region #2: base). The third (region#3: lower vertical region), fourth (region #4: middle vertical region), and fifth (region #5: upper verticalregion) phases were designed to provide information about the vertical regions of the object in terms ofsimilar information as before, and the sixth phase provided information about the rim or the top of theobject’s shape, size, texture, and material (region #6: rim). The last phase presented similar informationabout the inside of the object (region #7: inside). A fixed order of presentation during training greatlyfacilitates the training process and ensures generality of the approach.

Feedback was provided to the users on their recognition accuracy, and adjustments to the code weremade based on the user’s request. All 5 participants trained for half an hour with 5 objects as per thismethodology. A few notes on the training procedure: While our system allowed users to modify the cuesfor each feature, the participants did not change the cues suggested by our initial lexicon. Instead, theusers added cues for certain spatial features that they deemed salient. For example, one of the usersadded cues for conveying whether a given glass had a stem. This cue was presented on the middlefinger as a {‘dot,’ ‘dot,’ ‘dot’}. Another user added cues for conveying whether the object’s rim was largerthan the base or vice versa. These user-determined cues modeled the user’s value system pertaining tohaptics. An interesting development was that users often shared their cues and adopted cues developedby other users. This was very encouraging, as it depicted the ease of developing and customizing thecueing vocabulary, and the existence of shared concepts being communicated among users.

A series of experiments were designed to test if these simple tactile cues can invoke mental conceptsof objects. The first experiment involved the users receiving cues about an object’s features in a serialmanner for each phase. A fixed time gap of 1000 ms was inserted between consecutive cues. First, theshape cue is presented, followed by size, texture, and material in that order. In case the user has addedcues to the initial lexicon, these cues are presented after the initial cues. This is repeated for each ofthe 7 regions, as shown in Figure 7. Recognition accuracy of the cues was measured as the number ofcues that users correctly recognized, and for which they reported the correct feature value associated



with that cue. It is important to note here that recognition of cues requires the user to decipher whatfeature the user is perceiving (shape, texture, curvature etc.) and the quantization level for the feature.In our approach, if the user was not able to recognize either of the two parameters the cue presented,it was counted as a “miss.” If both the parameters were correctly recognized, a “hit” was recorded. Thisexperiment was conducted with 106 objects and the recognition accuracy was measured.

In the second experiment, we presented the users with a set of cues that conveyed features of anobject. Subsequently, users were presented with a set of 7 objects, and each user’s task was to identifythe object for which cues were presented. The object for which cues were presented was always includedin the 7 presented objects. Recognition accuracy was measured as the number of times the user correctlyidentified the objects as compared to total number of objects presented. This experiment was conductedwith 36 objects.

In the third experiment, we measured the effect of ambient and directed noise on the recognition ofobjects. We simulated ambient noise through an audio track that played in the background. Directednoise was simulated through the user being engaged in a conversation while receiving the cues. Theobjective of the user was to recognize the object. This experiment was conducted for all of the 5 userswith 36 objects. Twelve objects were presented in conditions of ambient noise, 12 objects were presentedin conditions of directed noise and 12 objects were presented in conditions of both directed and ambientnoise.

In the fourth experiment, we varied the time gap between consecutive cues. Time gaps were variedfrom 10 ms, 100 ms, 200 ms, 500 ms, 1000 ms, and 2000 ms. Five objects were presented in each of thetime gap categories. The task of the user in each time gap condition was to identify the object aboutwhich cues were presented, from a set of 7 objects. Recognition accuracy was measured.

In the fifth experiment, we tested the validity of spatial context. We changed the assignment offeatures to a scheme that was inconsistent with the neurophysiological specialization of the hand. Wetrained and tested the system (as per Experiment 1 defined above) with three such schemes and fiveusers. These three schemes randomly predetermined and assigned features to different finger feedbackunits. Using these schemes, five users who were sighted individuals blindfolded, were trained andtested.

In the sixth experiment, we tested the validity of temporal context. After the initial training, weallowed users to engage in interactive object exploration. We define interactive object exploration asthe process of command-response interactions between the user and the system wherein a user cancommand the system to present information about a feature through a hand gesture. The systemresponds with a haptic cue that is coded to represent the specific quantization level for the featurethe finger is associated with. This experiment explicitly models the user’s contextual model as it givesthe user control over what information is desired. The interactive methodology provides high flexibil-ity and dynamic adaptation that mimics the human reasoning process in real environments. Peopleoften employ contextual clues to locate objects and guess their identity. Visual clues, sensory infor-mation, and memory helps in locating and recognizing familiar and nonfamiliar stimuli. It is oftentedious to model such interactions. However, the interactive methodology allows users to guide thesystem based on what they deem necessary. This allows customization of the information flow fromthe system to the user, based on the current state of the user, his or her style, and strategy for rec-ognizing the stimuli. We wanted to test if users tend to follow their cognitive strategy during interactiveobject exploration, as that would demonstrate that our system closely mimics real environmentperception.

The following are the implementation details. Simple gesture recognition was employed to recognizewhich finger the user was bending. The feature associated with that finger’s feedback unit (for example,texture with the index finger) was presented. We divided the 5 users into 2 groups. Three users were



Fig. 8. Results for Experiments 1 and 2.

informed of their cognitive strategy of exploration as determined by the experiments described in theprevious section, and were requested to follow the cognitive strategy that they normally employ. Thisenabled us to inform users of their strategy and actively engage in object perception while following theirnatural tendencies. The other group of two users was asked to engage in interactive object explorationwithout any constraints. We wanted to see if this group a) performed better or worse than the first groupand b) if the group’s users automatically converge to their natural cognitive strategy of exploration.Thirty-six trials with different objects were conducted. We also measured how many cues were requiredto recognize the object, and the recognition rates.

5.1 Results

Figure 8 shows the recognition accuracy of Experiments 1 and 2. Figure 9 shows the results of Ex-periment 3. Figure 10 shows the results of Experiment 4. Figure 11 shows the results of Experiment5. Figure 12 shows the average number of cues required for recognition in Experiment 6. Each of theusers showed a 100% recognition accuracy during interactive object exploration. Figure 13 shows thestandard deviation of the users in group 2 from their cognitive strategy over the 36 objects that theyperceived.

5.2 Discussion of Results and Future Work

Results of Experiments 1 and 2 validate the hypothesis that simple contextual cues about object fea-tures presented through haptic cueing can invoke the mental image of the object for blind users. Almostall users had no problem in recognizing the cues and the attribute they convey. Object recognitionaccuracies while being very high, did not shows the same levels of accuracy. A possible explanationof this result is that for some objects, the cues are exactly the same and hence it is not possible forthe user to distinguish between the objects. A possible solution would be to provide realistic ren-dering after cueing, to convey all object features. This two stage rendering scheme may prove to bean effective methodology for haptic rendering. The results of Experiment 3 show that cueing is ro-bust even in the presence of noise. However, in the presence of both ambient and directed noise, theaccuracies were reduced. We intend to conduct more experiments with better simulations of noisyconditions.

The results of Experiment 4 show that users were comfortable recognizing objects at high speeds ofcueing. Interestingly, the results for object recognition using a time-gap of 2000 ms were not as highas other time intervals. We asked the users to subjectively evaluate the recognition results. In our



Fig. 9. Results for Experiment 3.


consultations, all the users expressed the fact that when the gap between cues is large, they tend toforget the previously presented cues. This raises an important question on the psychological basis ofthe cueing methodology. A possible explanation is that the human haptic perception system is tuned toreceive contextual spatial information within a certain range of intervals beyond which the piecemealinformation is not assembled into a complete entity. There is a significant body of work on workingmemory in psychology [Deutsch 1975]. We intend to explore this issue in more detail based on workingmemory and its principles.

The results of Experiment 5 show that spatial context plays an important role in our experiment.While the recognition accuracies of the cues dropped only by 5–8%, the time required for learning foreach of the schemes was significantly more than 30 minutes with the original formulation. We willrepeat this experiment with other formulations and measure the learning times.

Experiment 6 shows that users who were explicitly informed of their cognitive strategy (and hencehad knowledge of their temporal context) required fewer cues to recognize an object as compared to




Fig. 12. Average number of cues required for recognition in Experiment 6.

the other group of users. The results in Figure 13 show that both users who were interacting with thesystem without any constraints, naturally learned to follow their own cognitive strategies over certaintrials. This result further validates the significance of temporal context in haptic perception.

6. CONCLUSIONS

Haptic Perception in real environments is influenced by various factors [Revesz 1950]. In this article, weconducted experiments to model spatial, temporal, and user context of haptic perception in the hapticrendering process. Spatial context was modeled through presentation of haptic information to handsensors in coherence with their perceptual behavior. Temporal context was modeled as the cognitivestrategy individuals use to perceive objects. The user context was modeled as:

(1) modeling a user’s cognitive strategy to explore objects,

(2) modeling a user’s personal hand movement patterns for automatic recognition of exploratorygestures,



Fig. 13. Standard deviation of users in group 2 of Experiment 6 from their cognitive strategy in 36 trials of interactive

methodology.

(3) addition of user-specific cues to the presentation scheme,

(4) allowing interactive cue-based object exploration.

We designed a tactile cueing paradigm to test our models. Initial results show that modeling context inhaptic rendering environments improves perceptual accuracy even with a simple paradigm of contextualtactile cueing. To our knowledge, this is the first attempt at modeling context in haptic rendering; theresults indicate that encoding contextual information is a key to realistic haptic rendering. It ensuresthat users can perceive object features and identity with piecemeal information that is only a subsetof information available in the real environments. The proposed methodology for modeling contextmay help overcome the current limitations of haptic interfaces. While our methodology was testedfor haptic object recognition, contextual modeling can benefit other applications of haptic rendering,such as telesurgery, teleoperation, motor training, and so on. To further test our models, we proposeto incorporate them in veridical haptic rendering environments. A methodology to explicitly encodethe motor strategy of haptic exploration will be designed. We will test a proactive system against thebaseline haptic rendering system where the rendered objects and environments are static and do notadapt to a user’s state of exploration, or their temporal context.

REFERENCES

DEUTSCH, D. 1975. Short-term Memory. Academic Press, New York.

FRITZ, J. AND BARNER, K. 1997. Design of a haptic data visualization system for people with visual impairments. IEEETransactions on Rehabilitation Engineering 7, 3, 372–384.

HALE, K. S. AND STANNEY, K. M. 2004. Deriving haptic design guidelines from human physiological, psychophysical, and

neurological foundations. IEEE Comput. Graph. Applic. 24, 2, 33. 0272–1716.

HATWELL, Y., STRERI, A., AND GENTAZ, E. 2003. Touching for Knowing: Cognitive Psychology of Haptic Manual Perception.

Advances in Consciousness Research, v. 53. John Benjamins Pub., Amsterdam; Philadelphia. edited by Yvette Hatwell, Arlette

Streri, Edouard Gentaz. Cognitive psychology of haptic manual percetion. ill.; 23 cm.

JOHANSSON, R. 1996. Sensory control of dexterous manipulation in humans. In Hand and Brain: The Neurophysiology andPsychology of Hand Movements, H. P. Wing AM and F. JR, Eds. Academic, San Diego, 381–414.

KAHOL, K., TRIPATHI, P., MCDANIEL, T., AND PANCHANATHAN, S. 2005a. Modeling context in haptic perception, rendering and visu-

alization. In Lecture Notes in Computer Science: Proceedings of the 11th International Workshop, MIS 2005, ACM InternationalWorkshop on Multimedia Information Systems, K. Candan and A. Celentano, Eds. Vol. 1. Springer, Sorrento, Italy, 102–114.



KAHOL, K., TRIPATHI, P., AND PANCHANATHAN, S. 2005b. Haptic user interfaces: Design, testing and evaluation of haptic cueing

systems to convey shape, weight, material and texture information. In Human-Computer Interaction ISBN 0-8058-5807-5.

Lawrence-Erlbaum Associates, Las Vegas Nevada.

KAHOL, K., TRIPATHI, P., PANCHANATHAN, S., AND GOLDBERG, M. 2004. Formalizing cognitive and motor strategy of haptic ex-

ploratory movements of individuals who are blind. In IEEE International Workshop on Haptic Audio Visual Environmentsand their Applications. Vol. I. IEEE, Ottawa, Canada, 25–30.

KIPP, M. 2001. Anvil—a generic annotation tool for multimodal dialogue. In 7th European Conference on Speech Communica-tion and Technology. Aalborg, 1367–1370.

LEDERMAN, S. J. AND KLATZKY, R. 1987. Hand movements: A window into haptic object recognition. Cognitive Psychology 19,

342– 368.

LEDERMAN, S. J. AND KLATZKY, R. L. 1996. Action for Perception: Manual exploratory movements for haptically processing objects

and their features. In A. M. Wing and J. R. Flanagan, Eds., Hand and brain: The new-roophysiology and psychology of hand

movements. New York, Academic Press.

NIELSEN, L. 1995. Guiding hands: Help or hindrance? how the approach of guiding the hands of the visually impaired child

can disturb his opportunity to build up strategies for tactile orientation. Future Reflections 14, 3.

RABINER, L. AND JUANG, J. 1986. Introduction to Hidden Markov Models. IEEE ASSP Magazine, 4–16.

REVESZ, G. 1950. Psychology and Art of the Blind. Longmans Green, London.

SALISBURY, K., CONTI, F., AND BARBAGLI, F. 2004. Haptic rendering: introductory concepts. IEEE Comput. Graph. Applic. 24, 2,

24. 0272–1716.

TAN, H., GRAY, R., YOUNG, J., AND TRAYLOR, R. 2003. A haptic back display for attentional and directional cueing. Haptics-e 3, 1.

TZOVARAS, D., NIKOLAKIS, G., FERGADIS, G., MALASIOTIS, S., AND STAVRAKIS, M. 2004. Design and implementation of haptic virtual

environments for the training of the visually impaired. IEEE Transactions on Neural Systems and Rehabilitation Engineer-ing 12, 2, 266–278.

ZHONG, S. AND GHOSH, J. 2002. HMMs and coupled HMMs for multi-channel eeg classification. In IEEE International JointConference on Neural Networks.

Received May 2006; accepted May 2006


Documents

Modeling context in haptic perception, rendering, and visualization