Haptizing Surface Topography with Varying StiffnessBased on Force Constancy: Extended Algorithm

Jaeyoung Cheon∗ Inwook Hwang† Gabjong Han‡ Seungmoon Choi §

Haptics and Virtual Reality LaboratoryDepartment of Computer Science and Engineering

POSTECH, Republic of Korea

ABSTRACT

This article introduces a novel haptization method for renderingsurface topography with varying stiffness via a force-feedback hap-tic interface. Previously, we showed that when surface topographywith varying stiffness is rendered with the conventional penalty-based method, topography information perceived by the user canbe distorted from its model. This phenomenon was explained bythe theory of force constancy which states that the user maintainsan invariant contact force level when s/he strokes a surface to per-ceive its topography. To resolve the problem, we then developeda basic topography compensation algorithm (TCA) based on theforce constancy, for a single height-change region with nonuniformstiffness perceived via lateral stroking. The basic TCA was mainlyto test the applicability of force constancy to haptic rendering. Inthis article, we present an extended TCA that adequately deliverssurface topography that may contain a number of height-changingregions with varying stiffness for any user exploratory patterns. Wealso measured the human detection thresholds of surface slope me-diated with a force-feedback device and used these data for design-ing the extended TCA. The performance of the extended TCA wasextensively examined in terms of proximal stimuli it creates and ac-tual percepts induced from the stimuli. The extended TCA bringsa one-step advance from the current practice of haptic renderingwhich requires constant surface stiffness for an adequate deliveryof surface shape.

Index Terms: H.5.1 [Information Interfaces and Presentation]:Multimedia Information Systems—Artificial, Augmented, and Vir-tual Realities; H.5.2 [Information Interfaces and Presentation]:User Interfaces—Haptic I/O

1 INTRODUCTION

Data perceptualization aims at delivering the properties of a data setto the user through multi-modal sensory channels and is a promis-ing application area of haptics. Extended from traditional data vi-sualization, other sensory modalities such as sound (data sonifica-tion) and touch (data haptization) are actively involved in data per-ceptualization, so that the user can see, hear, and touch the datawith increased information transfer bandwidth. By perceptualiza-tion techniques (or transfer functions), a data variable (e.g., densityat a voxel) is mapped to a display attribute (e.g., color, opacity,or force, etc.). A critical requirement here is that information per-ceived by the user must agree with the original information con-tained in a data set. Perceptualization methods that fail to ade-

∗e-mail: icejae02@naver.com. Currently at the Agency for Defense De-velopment, Republic of Korea.

†e-mail: inux@postech.ac.kr.‡e-mail: hkj84@postech.ac.kr§e-mail: choism@postech.ac.kr.

quately take into account the user’s resulting percepts may causeincorrect understanding of data properties by the user.

A pioneering work of data perceptualization including the senseof touch was the GROPE project that allowed users to perceive andcontrol the docking of molecules via a force feedback device and ahead-mounted visual display [2]. Shortly after that, Taylor et al. de-veloped a teleoperation system that interfaced a force-feedback de-vice to a scanning tunneling microscope, providing both visual andhaptic feedback of measured data properties [23]. Then, researchinterests shifted to haptic volume rendering algorithms that used thelocal gradient to determine force transfer functions [12][1]. The tra-ditional proxy-based haptic rendering technique was also extendedto volumetric data rendering [16][11]. Recently, Maciejewski et al.released a haptization technique for multiple volumetric data sets[17]. In an initial attempt to simultaneously haptize multiple hap-tic attributes (surface shape and stiffness), Yano et al. showed thatperceived stiffness distribution on a flat surface may not match toits model and proposed a simple compensation technique that pre-warps the stiffness model [26].

Object shape and stiffness are the two most fundamental hapticproperties. Most virtual reality applications emulating real envi-ronments presume constant stiffness for an object/surface. In dataperceptualization, however, stiffness can be another continuouslyvarying data attribute. One such example is provided in Figure 1that shows a surface height map of bi-lipid membrane patches withembedded proteins on a mica substrate, with brighter colors cor-responding to higher regions. The topography map was acquiredwith a scanning probe microscope that can measure various collo-cated nanometer-scale features on a flat substrate [22]. The imageshows protein membrane patches surrounded by a halo (presum-ably of lipids that have dissociated from the membrane) resting onthe atomically flat mica substrate. Because the membrane patch isfilled with a periodic array of the transmembrane protein, it shouldbe considerably stiffer than the halo of dissociated lipids, but not asstiff as the mica substrate. The surface height profile of the data setis illustrated in Figure 2 with a solid line. For convenience, we willrefer to the data set as the “protein-on-mica” data set.

At first, we attempted to render the “protein-on-mica” data setwith the traditional penalty-based haptic rendering technique (e.g.,[27][20]). A user could tap on and stroke the resulting virtual hap-tic surface to gauge the local stiffness and topography, respectively.However, the halo regions were consistently felt lower than the sur-rounding mica regions in opposite to the height model. This obser-vation was confirmed through probe-tip trajectories recorded duringa user’s stroking on the virtual surface. A typical probe-tip tracesimplified for illustration is shown in Figure 2 with a dotted line,which indicates that the probe tip was clearly “dipped” into the haloregion against the height map.

This undesired result of haptization was proven in our follow-ing study due to the force constancy, a human exploratory behaviormeaning that the user tends to maintain a constant contact forcewhile stroking a virtual surface to perceive its shape [7]. This ex-plains why the halo region with lower stiffness in Figure 2 allows

193

Symposium on Haptic Interfaces for Virtual Environments and Teleoperator Systems 200813-14 March, Reno, Nevada, USA978-1-4244-2005-6/08/$25.00 ©2008 IEEE

Figure 1: The surface height map of “protein-on-mica” data. Higherregions are coded with brighter colors.

Figure 2: A cross-section of the height map shown in Figure 1 alongwith the trajectory of a haptic probe tip.

larger probe penetration than the mica with higher stiffness. If apenetration depth increase exceeds the modeled height differencebetween the halo and mica regions, the halo region is perceived tobe lower than the mica region as shown in Figure 2. The theory offorce constancy was also confirmed in [5] recently.

We then tested the applicability of the force constancy theory tohaptizing surface topography with varying stiffness using only thelimited information available during haptic rendering [3]. A sim-ple domain adequate for the purpose was chosen: a surface with asingle height-changing interval with different stiffness values per-ceived via lateral stroking1 (e.g., see Figure 3). We developed abasic Topography Compensation Algorithm (TCA) that estimates aforce applied to the surface by the user using previous force com-mands sent to a haptic interface and then adaptively adjusts a sur-face height profile based on the force constancy theory. Renderedforces had a constant direction orthogonal to a baseline surface(e.g., along the normals of planes P1 and P2 in Figure 3) to focuson the role of kinesthetic sensory cues resulted from a normal po-sition change of the probe tip. It was confirmed that the basic TCAcreates appropriate kinesthetic sensory cues and is quite effective atrendering relatively large surface height changes in a perceptuallycorrect manner.

In a next study [4], we investigated the advantages of addingsurface normal rendering to the basic TCA. By changing renderingforce directions according to the surface geometry (e.g., along thenormals of the surface between P1 and P2 in Figure 3), additionallateral force cues can be generated when the probe tip crosses aheight changing region. Surface normal rendering was proven togreatly facilitate the user’s perception of a surface height change,especially for those too small to be reliably discriminated with thekinesthetic cues alone (< 1 mm).

In this article, we present an extended TCA that no longer re-quires the assumption made in the basic TCA (Section 2). Theextended TCA can properly render surface topography that maycontain a number of height-changing regions with varying stiffnessfor any user exploratory patterns. The extended TCA has two majorimprovements compared to the basic TCA. First, the extended TCAis capable of canceling out compensation errors that can be accu-

1The exploratory procedure refers to stereotypical hand movement pat-terns employed by the humans to haptically perceive objects attributes. Forexample, “tapping” to feel the stiffness of a surface, and “stroking” to feelthe shape and texture of the surface. See [13][18] for further details.

Figure 3: Top view of the haptic rendering of two vertical planes. Thesymbol at the bottom represents a user facing the vertical surface.

mulated during a prolonged use of the basic TCA. A rule for errorcancellation is carefully designed based on the human sensitivityto surface height change (i.e., slope) measured in a psychophysi-cal experiment (Section 3). Second, an adequate collision detectionstrategy is incorporated in the extended TCA to take care of any ex-ploratory procedures for all possible configurations of the tool-tipposition and adjusted surface height. We extensively examined theperformance of the extended TCA, in terms of proximal stimuli itcreates (Section 4) and actual percepts it induces (Section 5), andconfirmed that the extended TCA can deliver both surface topogra-phy and stiffness consistently with their models in a data set. A planfor future work is also provided along with conclusions (Section 6).

2 EXTENDED TOPOGRAPHY COMPENSATION ALGORITHM

This section explains the extended TCA in details. Let n be atime index for haptic update loops. We denote a haptic interfacepoint (HIP; the probe-tip position of a haptic interface) by p(n) =(px(n), py(n), pz(n)), a surface contact point (SCP; an avatar ofthe HIP constrained on a surface for rendering force computation[10], also called the god object [27] or the virtual proxy [20]) byq(n), and a compensated surface contact point (CSCP; an adjustedsurface contact point for topography compensation) by q′(n) (seeFigure 4). Without loss of generality, we assume that a heightmap h(x,y) and a stiffness map k(x,y) are defined on the xy planeas in Figure 1. We also denote that h(n) = h(px(n), py(n)) andk(n) = k(px(n), py(n)) for simplicity. Since the SCP q(n) is alwayson the surface, we can also find an outward normal vector n(q(n))at q(n). A force command to be sent to a haptic interface is repre-sented by f(n).

A pseudo code of the extended TCA is shown in Algorithm 1.Assuming no collision between the HIP and the surface model inthe beginning of rendering, the extended TCA initializes variablesas: COLLISION← FALSE and Sh(0)← 0. A flag variable COLLI-SION indicates whether a collision between the HIP and a surfaceadjusted by the CSCP occurred in the previous rendering frame.Sh(n) represents the total amount of surface height compensationto be added to the modeled height.

Given the HIP p(n), line 1 in the pseudo code obtains the SCPq(n) and its normal n(q(n)). This is supported in most collisiondetection packages for haptic rendering, and our implementationused a method in [21] for implicit models. In lines 2–6, if therewas no collision in the previous frame, a height compensation termΔhc(n) is set to zero. Otherwise,

Δhc(n)=

⎧⎨⎩

0 if k(n) = k(n−1)

fz(n−1)(

1k(n)

− 1k(n−1)

)otherwise

.

(1)The amount of necessary height compensation is determined basedon the force constancy theory using information attainable duringhaptic rendering. According to the force constancy [7], the amountof height distortion due to different stiffness values is:

Δhd(n) =− fp(n)(

1k(n)

− 1k(n−1)

), (2)

194

Algorithm 1 Extended Topography Compensation Algorithm.

1: Given the HIP p(n), find the SCP q(n) and its normal n(q(n)).2: if COLLISION = TRUE then3: Compute a height compensation term Δhc(n) in Equation 1.4: else5: Δhc(n)← 06: end if7: Compute a height restoration term Δhr(n) in Equation 3.8: Compute an accumulated height change Sh(n) in Equation 4.9: Compute a CSCP q′(n) in Equation 5.

10: if (q′(n)−p(n)) ·n(q(n)) > 0 then11: Compute a rendering force f(n) in Equation 6.12: COLLISION← TRUE13: else14: f(n)← 015: COLLISION← FALSE16: end if17: return f(n)

where fp(n) is the force applied by the user. In our design of Δhc(n)to compensate Δhd(n), the user-applied force at n is estimated as thedevice force command at n−1. This is based on the facts that thehuman motion bandwidth is much lower than the haptic renderingupdate rate (thus fp(n) � fp(n− 1)) and the device commandingforce should be very similar to the user-applied force due to theforce constancy (thus fp(n−1) � fz(n−1)).

A next step in line 7 is to compute a height restoration termΔhr(n) as:

Δhr(n) =

⎧⎨⎩−Sh(n−1) if |Sh(n−1)| ≤ |ar dxy(n)|−ar dxy(n) if Sh(n−1) > ar dxy(n)ar dxy(n) if Sh(n−1) <−ar dxy(n)

, (3)

where dxy(n) = ‖(px(n)− px(n−1), py(n)− py(n−1))‖ is the lat-eral displacement of the probe-tip and ar is a slope for heightrestoration.

The height compensation term Δhc(n) was shown to be quite ef-fective at properly delivering a single height change with varyingstiffness [3]. However, the estimation of a user-applied force inEquation 1 includes inevitable errors. These errors, once accumu-lated in Sh(n), may cause undesirable behaviors such as one illus-trated in Figure 5. This data was captured when a user repeatedlystroked a height changing region (e.g., one shown in Figure 3) ren-dered with the basic TCA (i.e., without Δhr(n)) back and forth (e.g.,from left to right, then right to left). The height change Δh was 3mm and its width W was 4 mm. The stiffness values of the left andright planes were k1 = 0.9 and k2 = 0.3 N/mm, respectively. Theadjusted height model using the basic TCA (see q′z(n) in the up-per panel of Figure 5) continued to be elevated during the repeatedstroking, which made the normal position of the probe-tip (pz(n))increase as well. This was due to the values of Δhc(n) that were notsymmetrical (see the lower panel of Figure 5), with those of Δhc(n)in the left-to-right strokes larger those in the right-to-left strokes,and these differences were accumulated in Sh(n) over time.

In the extended TCA, the two terms, compensation term Δhc(n)and restoration term Δhr(n), compete each other. The compensa-tion term makes the CSCP deviate from the SCP based on the forceconstancy theory to render height changes consistently with a sur-face topography model. The restoration term enforces the CSCP toconverge to the SCP by removing accumulated errors in the totalheight adjustment due to the inaccuracy of user-applied force es-timation. A crucial requirement here is that the rate of change inthe restoration term must be “slow,” so that it does not adverselyaffect the proper height compensation. To have a reference for this,we measured the smallest amount of surface slope that can be per-

Figure 4: Illustration of the haptic interface point (HIP), surface con-tact point (SCP), and compensated surface contact point (CSCP).

0 1000 2000 3000 4000 5000

0

10

20

30

Time (ms)

Hei

ghts

(m

m)

0 1000 2000 3000 4000 5000−1

0

1

Time (ms)

Δ h c (

mm

)

h(n) pz(n) q’

z(n)

Figure 5: Error accumulation in the basic TCA.

ceived by the user (see Section 3). Several values of ar selectedbased on the slope detection thresholds were experimentally testedin a psychophysical experiment (see Section 5). Note that the sur-face height is restored until Sh(n) = 0 even if there was no collisionin the previous frame. Resetting Sh(n) to 0 right after the contactis broken may cause the following problem: When a user scans thesurface in Figure 4 from left to right, lifts the probe above the ad-justed height (represented by the CSCP), and then pushes the probeback to the surface, the user may notice the difference between theprobe positions at the moments of leaving and touching the surfaceback.

A next step, line 8 of the pseudo code, is to compute a totalamount of height change Sh(n) by accommodating the previous andcurrent height compensation and restoration terms as follows:

Sh(n) = Sh(n−1)+Δh(n)+Δhc(n)+Δhr(n) , (4)

where Δh(n) = h(n)−h(n−1) is the modeled height difference.Then, in line 9, the CSCP q′(n) is determined using Sh(n) and

the current SCP q(n) as:

q′(n) = q(n)+Sh(n)q(n)−p(n)‖q(n)−p(n)‖ . (5)

This places the CSCP at an elevated (or dropped) position from theSCP by Sh(n) in the direction from the HIP to the SCP. When theuser strokes the surface in Figure 4 from left to right, the renderingforces within the height change interval oppose the movement ofthe probe tip, creating the sensation of lateral contact. When theprobe tip moves from right to left, the rendering forces within theheight change interval accelerate the probe tip to the lower plane.This provides additional lateral force cues for height change per-ception.

The next line 10 in the pseudo code is for collision detectionwhich is another issue that has to be carefully handled in the ex-tended TCA. For example, consider a scenario that the user verti-cally lifts the probe from the HIP position shown in Figure 4 after

195

Figure 6: Possible configurations of the HIP, SCP, and CSCP.

stroking the surface from left to right. Comparing the HIP to theoriginal surface model would declare no collision as soon as theHIP is elevated over the height model on which the SCP is con-strained. However, since rendering forces are to be computed basedon the distance between the CSCP and the HIP (see Equation 6),this would result in a discontinuity in the rendered force.

Total six relative configurations are possible among the HIP,SCP, and CSCP as shown in Figure 6. In cases 1, 2, and 3 in the fig-ure, the HIP is under the CSCP and a force should be rendered. Thiscondition can be expressed as a vector from the HIP to the CSCP,q′(n)−p(n), is in the same direction with the outward surface nor-mal at the SCP, n(q(n)). If the condition is satisfied, a renderingforce command f(n) is calculated using the CSCP instead of SCPin line 11 of the pseudo code as:

f(n) = k(n)(q′(n)−p(n)

). (6)

This rendering force provides two kinds of sensory cues: the kines-thetic cue resulted from the height change of the probe tip and thelateral force cue induced from the varying rendering force direc-tions. The flag variable COLLISION is also set to TRUE for a nextframe. If the condition in line 10 is not satisfied (cases 4, 5, and 6in Figure 6), no forces are rendered in line 14 and COLLISION isset to FALSE in line 15.

With an adequate choice of ar , the extended TCA restores theCSCP to be on the original surface after the rise (or fall) of theprobe tip in a height-changing region with varying stiffness, so thatthe CSCP would be equal to the SCP. This would create a short“pulse” to the probe-tip and deliver kinesthetic cues helpful for thediscrimination of the height difference (e.g., see Figure 9). Percep-tual performance of the extended TCA was validated in experimentsreported in Section 4 (probe trace measurement) and 5 (user dis-criminability measurement). Computationally, the extended TCAis very efficient and requires no preprocessing of the models.

Note that if k(n) in Equation 2 approaches to 0, the height com-pensation term Δhc(n) can be arbitrarily large. This may impedethe proper height restoration by Δhr(n), but can be prevented inpractice by appropriately defining the range of surface stiffness forrendering mapped from that of original stiffness in a scientific dataset.

3 EXPERIMENT I: HAPTIC DETECTION OF SURFACE SLOPE

The purpose of Experiment I is to find the smallest surface slopethat can be reliably perceived by the user when the surface isrendered with a force-feedback haptic interface. This experimentwas motivated to design an adequate height restoration term tobe used in the extended TCA. Although there have been studieson the perception of object shape and surface curvature (e.g., see[24][9][15][25]), we were unable to find one that can be directlyused in the extended TCA. The results of this experiment also haveimplications on designing haptic watermarks to be embedded in 3Dmesh models [19].

Figure 7: Visual scene provided to the subjects in Experiment I.

3.1 Methods3.1.1 Apparatus

As a force-feedback device, a PHANToM (model 1.0A; SensableInc.) was used to render virtual surfaces used in the experiment2.A 19-inch LCD monitor provided visual scenes for the experiment.

3.1.2 Subjects

Ten subjects (S1 – S10; 20 – 25 years old with the average of 21.9)participated in the experiment. S1 – S5 were females, and S6 –S10 were males. All subjects except for S3 were right-handed byself-report. S1, S3, and S7 had experiences of participating hapticperception experiments prior to this experiment, and S5 was an ex-perienced user of a force-feedback device. All other subjects hadnot been exposed to haptic interfaces before this experiment. Nosubjects reported any known sensorimotor abnormalities.

3.1.3 Stimuli

The haptic stimulus presented via the PHANToM was a verticalvirtual plane slanted with a certain slope. Specifically, the heightmodel of the plane was:

hz(n) = a dxy(n), (7)

where hz(n) is the plane height and a is the plane slope (see Figure3 again for the coordinate frame). Note that a = 0 renders the non-slanted vertical plane facing the subject. The force commands sentto the PHANToM were computed using the usual penalty-basedrendering with a stiffness parameter k.

The visual scenes provided to the subjects (see Figure 7) in-cluded an interval number, four regions with different colors to aidstroking movement (to be explained soon), and a cone representingthe HIP. No visual information for plane slope was displayed.

3.1.4 Experimental Conditions

Four independent variables were defined for the experiment. Thefirst variable was whether the plane became higher or lower duringleft-to-right stroking (i.e., the sign of a in Equation 7). The sec-ond one was surface stiffness: 0.9 N/mm for a hard surface and0.3 N/mm for a soft one. Note that with the PHANToM, surfaceswith stiffness values larger than 1.0 N/mm become unstable, whilestiffness values less than 0.3 N/mm are too mushy to clearly definea surface [6]. The third one was the width of a height-changing

2After this paper was accepted, we found that the PHANToM used forall experiments reported in this paper was malfunctioning, producing onlythe half of force command. This was not noticed until we attached a forcesensor for another application. It was due to a fault in its communicationboard (personal communication with the manufacturer). Thus, actual stiff-ness values used could have been the half of those reported in this paper,but we cannot ascertain it for now. We will redo all experiments and postrevised results in our web site at http://hvr.postech.ac.kr.

196

Table 1: Experimental conditions used in Experiment I.

Experimental ConditionC1 C2 C3 C4 C5 C6 C7 C8 C9 C10 C11 C12 C13 C14 C15 C16

Slope Direction ↗ ↘Stiffness (N/mm) 0.9 0.3 0.9 0.3

Region Width (mm) 100 20 100 20 100 20 100 20Stroking Velocity (mm/s) 100 20 100 20 100 20 100 20 100 20 100 20 100 20 100 20

interval (100 mm and 20 mm). The last independent variable wasstroking velocity, 100 mm/s for fast movement and 20 mm/s forslow movement. All of the independent variables seemed to af-fect the detection thresholds of plane slope in pilot experiments.As combinations of the four variables, 16 experimental conditionswere defined as summarized in Table 1.

3.1.5 Procedures

The three-interval, forced-choice, one-up three-down adaptive stair-case method [14] was used. In this method, three consecutive cor-rect responses lead to a reduction in stimulus intensity and one in-correct response leads to an increase, both by a predefined step size.This combination yields the 79.4 percentile point on a psychomet-ric function. Each trial consisted of three intervals where only onerandomly selected interval rendered a slanted plane and the othertwo rendered the plane with zero slope.

The subject sat comfortably in front of the LCD monitor hold-ing the PHANToM stylus with the right hand. The responses of thesubject were typed in with the left hand using a keyboard. In thebeginning of each interval, the subject moved the PHANToM sty-lus onto the red rectangle on the left end (see Figure 7) and pressedthe space bar in the keyboard to initiate the interval. This wouldmove the blue rectangle from left to right with the stroking veloc-ity predefined in the experimental condition to provide a referencefor stroking velocity to the subject. The subject was instructed tofollow the blue rectangle while keeping a contact with the plane,until reaching the green rectangle on the right end. This completedthe subject’s task in the interval. If the HIP fell outside the bluerectangle during stroking, the interval was discarded and repeatedagain. After the trial, the subject was asked to answer in which ofthe three intervals a plane felt slanted by pressing a correspondingkey (‘1’, ‘2’ or ‘3’) on the keyboard.

An initial slope for each experimental condition was chosen tobe much higher than the expected detection threshold level foundin the pilot experiments. The step size was initially set to 4 dB (forfaster convergence) and then reduced to 1 dB (for finer resolution)after the first three reversals (a reversal occurred if the slope valuechanged from increasing to decreasing, or vice versa). A series oftrials was terminated after 12 reversals at the 1-dB step size. Nocorrect-answer feedback was provided to the subjects.

Before the experiment, the subject went through training ses-sions, especially to become familiar with following the blue rect-angle during stroking, under four conditions differing in height-changing region width and stroking velocity. It took 5–15 minutesto finish one experimental condition, and each subject completed5–6 experimental conditions per day. The whole experiment pro-ceeded in three sessions, one per each day.

3.1.6 Data Analysis

In each experimental condition, the slope values where the last 12reversals (six peaks and six valleys) at the 1-dB step size occurredwere recorded and averaged pairwise, resulting in six estimates ofa slope detection threshold. The six estimates were used for thecomputation of a final threshold average and a standard error.

C1 C2 C3 C4 C5 C6 C7 C8 C9 C10C11C12C13C14C15C160

0.05

0.1

0.15

0.2

0.25

0.3

Experimental Condition

Det

ectio

n T

hres

hold

of S

urfa

ce S

lope

Figure 8: Detection thresholds of the plane slant perception.

3.2 Results

The measured detection thresholds of surface slope averaged acrossthe subjects are shown in Figure 8, along with error bars repre-senting the standard errors. Overall, the detection thresholds var-ied in 0.09 (in C1) – 0.25 (in C16). The detection threshold un-der each experimental condition corresponds to a ratio betweenthe vertical and lateral displacements of the probe tip for the ver-tical change to be reliably perceived. The effects of the four in-dependent variables were tested via a four-way ANOVA with thedetection threshold as a dependent variable. It was shown that stiff-ness and height change width had statistically significant influenceson the slope perception thresholds (F(1,9) = 52.24, p < 0.001 andF(1,9) = 40.70, p < 0.001, respectively), but height change direc-tion and stroking velocity did not (F(1,9) = 1.47, p = 0.2571 andF(1,9) = 0.89, p = 0.370, respectively).

The data in Figure 8 suggest a few guidelines for selecting thevalue of ar that determines the rate of height restoration in the ex-tended TCA. If values less than 0.09 (the smallest slope detectionthreshold) are used for ar, it is unlikely that the user can perceive theeffect of height restoration. If values higher than 0.25 (the largestslope detection threshold) are used, the user may feel its effect, butthis may be masked by the more abruptly changing height com-pensation term. The results of this experiment provide an intervalfor suitable candidates of ar, and several values of ar were furthertested in another psychophysical experiment in Section 5.

4 EXPERIMENT II: PROBE TRAJECTORIES

In this section, we present proximal stimuli perceived by the userwhile exploring virtual surfaces with varying stiffness rendered us-ing the extended TCA. In our algorithm, a force vector directs fromthe HIP p(n) to the CSCP q′(n) and its magnitude is proportionalto the distance between them. Due to the limited space, we mostlyprovide the traces of the HIP and CSCP from which rendered forcemagnitudes and directions can be easily extracted.

The data in the upper panel of Figure 9 show the normal coor-dinates of the HIP and CSCP (pz(n) and q′z(n), respectively) col-lected when a user stroked from left to right a surface modeled withk1 = 0.9 N/mm, k2 = 0.3 N/mm, Δh = 3 mm, and W = 4 mm. The

197

1500 2000 2500 3000−5

0

5

10

Time (ms)

Hei

ghts

(m

m)

1500 2000 2500 3000−5

0

5

10

Time (ms)

Hei

ghts

(m

m)

h(n) pz(n) q’

z(n)

Figure 9: Traces of the HIP, SCP, and CSCP during one-time stroking.

0 1000 2000 3000 4000 5000−10

0

10

20

Time (ms)

Hei

ghts

(m

m)

0 1000 2000 3000 4000 5000−10

0

10

20

Time (ms)

Hei

ghts

(m

m)

h(n) pz(n) q’

z(n)

Figure 10: Traces of the HIP, SCP, and CSCP during repeated strokes.

0 1000 2000 3000 4000 5000−100

0

100

Time (ms)

p x(n)

(mm

)

0 1000 2000 3000 4000 5000−10

0

10

20

Time (ms)

Hei

ghts

(m

m)

h(n) pz(n) q’

z(n)

Figure 11: Traces of the HIP, SCP, and CSCP for two consecutive ex-ploratory procedures: stroking and then tapping.

−60 −40 −20 0 20 40 60−5

0

5

10

x (mm)

h(x)

(m

m)

0 1000 2000 3000 4000 5000−5

0

5

10

Time (ms)H

eigh

ts (

mm

)

h(n) pz(n) q’

z(n)

MicaHalo

Protein

Figure 12: Traces of the HIP, SCP, and CSCP for surface and stiffnessmodels representing the “protein-on-mica” data.

two planes were smoothly connected via Hanning windows (half-cycle sinusoidal functions) in both height and stiffness (see [7] forthe exact models). Note that this parameter set is an extreme case interms of stiffness difference and was purposely chosen to demon-strate how the height compensation and restoration work in the ex-tended TCA. The same parameter set was used for the data shownin Figures 9 – 11. For the lower panel in Figure 9, k1 and k2 wereswitched with Δh =−3 mm. The height restoration slope ar was setto 0.10. In both plots, a short position “pulse” of about 500 ms dura-tion, which aids the correct perception of surface height, is observedin 2000 – 2500 ms, and the CSCP is reverted back to the SCP afterthe pulse. This normal movement for restoration also brings sen-sory cues for the perception of surface stiffness, and most partici-pants in Experiment III (in the next section) reported that they couldfeel height difference as well as stiffness change during stroking.

Figure 10 confirms that errors accumulated in the total heightcompensation Sh(n) in the basic TCA (e.g., see Figure 5) are nolonger present in the extended TCA. The data were collected whilea user repeatedly scanned the region of a height change back andforth. The upper panel is for the case where the right plane washigher and stiffer with the same parameters used in Figure 9. Thelower panel is for the other case. For height restoration, ar was setto 0.05 that was less than the smallest detection threshold measuredin Experiment I. Despite such small value of ar under the largestiffness difference, one can see that the surface height was properlycompensated without any excessive error accumulation.

Figure 11 demonstrates how the extended TCA behaves for ex-ploratory procedures other than stroking. Here, the upper panelshows the probe-tip position in the stroking direction, px(n), andthe lower panel the normal positions of the HIP and CSCP. ar was0.10 for this data. The user stroked the surface from left to right in 0– 1500 ms, went over the height change interval in 1500 – 3000 ms,

lifted the stylus vertically in 3000 – 3500 ms, and pushed it backto the surface in 3500 – 4500 ms. Our collision detection strategydoes not accompany any force discontinuities, and thus no abruptposition movements are observed in the HIP and CSCP positionswhen the contact was broken during the stylus lifting at about 3200ms. Also note that when the stylus came back to the surface fortapping at about 3700 ms, the value of q′z(n) was almost the sameas that when the stylus left the surface, providing the consistent po-sition reference to the user. Although the contact had a slight offsetabout 1 mm above the surface model, such small offsets are hardlynoticeable haptically. Even if a visual representation of the surfaceis provided, such differences are hindered by the usual volumetricrepresentation of the HIP (e.g. a sphere or a cone). Also recall thatthe stiffness difference in this example is very large. Such offsetsdo not appear in most of other cases with less stiffness differenceswhere the CSCP converges to the SCP more quickly.

Finally, an example where the extended TCA (ar = 0.10) wasapplied to topography and stiffness models similar to the “protein-on-mica” data set is shown in Figure 12. The upper panel showsthe height model. The stiffness was set to 0.9 N/mm for the micaand protein regions and to 0.3 N/mm for the halo, with Δh = 0.3mm and W = 4 mm in all height-changing regions. A user scannedthe model from left to right. The lower panel shows that the shortpulse peaked at about 1600 ms clearly delivered the higher positionof the mica region and the-probe tip drop in 1600 – 2000 ms al-lowed the user to perceive the lower stiffness of the mica. Similarobservations can be made for the other height-varying regions.

5 EXPERIMENT III: HEIGHT DIFFERENCE PERCEPTION

The extended TCA includes the height restoration term Δhr(n) thatreduces the effect of height compensation for error correction. De-pending on the rate of height restoration, it may prohibit delivering

198

surface topography accordingly to the surface model. In this exper-iment, we investigated how different values of restoration slope arcan affect height discriminability in the extended TCA.

5.1 Methods5.1.1 Apparatus

The same equipments as Experiment I were used.

5.1.2 Subjects

Eight subjects (S1 – S8) participated in this experiment. S3 andS4 were females, and the others were males. Their ages ranged in19 – 26 years with the average of 21.9. All subjects were right-handed by self-report. No subjects had participated in any hapticperception experiments prior to the present experiment and reportedany known sensorimotor abnormality.

5.1.3 Stimuli

The haptic stimuli used in the experiment consisted of two verti-cally adjoined planes facing the user as shown in Figure 3. Thetwo planes were smoothly connected via Hanning windows in bothheight and stiffness. The stiffness values were k1 = 0.7 N/mm andk2 = 0.4 N/mm. Rendering forces were computed with the ex-tended TCA in all experimental conditions.

During the experiment, a LCD monitor displayed visual informa-tion only to provide a spatial reference to the subjects. The visualscene consisted of two blocks representing the starting and endingregions for stylus stroking and a blue cone showing the stylus-tipposition (see [3] for details).

5.1.4 Experimental Conditions

Three height offsets (Δh = 0.6, 0.8, and 1.0 mm) were tested. Thesesmall height differences were close to the previously published de-tection threshold of surface height change detection (� 0.66 mm3),and were reliably discriminated with the basic TCA [4]. In pi-lots, we confirmed that the extended TCA robustly delivers heightchanges larger than 1 mm and chose the most challenging heightdifferences. Three slope values (ar = 0.05, 0.10, and 0.30) wereused for height restoration. ar = 0.10 was chosen to be close to thesmallest detection threshold of slope perception measured in Exper-iment I, ar = 0.05 to be a more conservative value, and ar = 0.30to be slightly larger than the largest detection threshold. Combin-ing the two independent variables, there were total 9 experimentalconditions (3 height changes × 3 height restoration slopes).

5.1.5 Procedure

One-interval two-alternative forced-choice paradigm [8] was used.In one alternative (P1→P2), a lower and stiffer plane (P1) was pre-sented on the left, and a higher and softer plane (P2) was on theright. In the other case (P2→P1), P2 was on the left, and P1 wason the right. Each alternative was chosen with equal probability ineach trial. The order of experimental conditions was randomizedfor each subject. In each trial, the subject was asked to stroke a vir-tual surface once from left to right and to answer which plane felthigher than the other. Stroking velocity was not controlled in thisexperiment. We also recorded probe-tip trajectories and renderedforces. See [3] for further details about the procedure.

Each experimental condition consisted of 100 trials, and tookapproximately 10 minutes to be completed. The subjects were re-quired to take a break for five minutes after finishing each exper-imental condition. Each subject completed the experiment in twodays (4 – 5 experimental conditions per day). No correct-answerfeedback was provided to the subjects. The training procedure ofthe subject was the same as that of [3].

3This detection threshold was measured with surface models of constantstiffness and constant direction forces [7]. Note that it was for nominalsurface height difference (Δh), not actual height difference (Δpz).

Ca Cb Cc−5

−4

−3

−2

−1

0

1

2

3

4

5

Experimental Condition

d’

Δh = 0.6 mmΔh = 0.8 mmΔh = 1.0 mm

Figure 13: Average d′ values measured in Experiment III. Ca: ar =0.05. Cb: ar = 0.10. Cc: ar = 0.30.

Table 2: Mean actual height differences measured in Experiment III.

Δpz (mm)P1→ P2 P2→ P1

Δh (mm) Ca Cb Cc Ca Cb Cc0.6 0.45 0.37 -0.33 -0.14 0.05 0.690.8 0.57 0.45 -0.03 -0.32 -0.03 0.551.0 0.75 0.60 0.38 -0.32 -0.17 0.46

5.1.6 Data Analysis

The sensitivity index (d′) was used for a measure of discriminabil-ity independent from response bias [8]. In our setting, a positive d′value indicates that P2 was perceived to be higher than P1, consis-tently to the models. A d′ value close to zero implies that P1 andP2 were perceived to be of equal height. A negative d′ value meansthat P2 was perceived to be lower than P1, against to the models.

The actual height differences in the probe-tip position, Δpz,were also estimated from the recorded probe-tip trajectories. Weaveraged the probe-tip heights in two intervals, one on the leftplane (px(n) ∈ [−10mm,−2mm]) and the other on the right plane(px(n) ∈ [2mm,10mm]). Recall that the height change occurred inthe 4 mm interval centered at x = 0). We then subtracted the aver-age probe-tip height of the left plane from that of the right plane toobtain Δpz values.

5.2 Results and DiscussionThe d′ values averaged across all subjects are shown in Figure 13for all experimental conditions. All d′ values were much largerthan 1, indicating that the subjects reliably discriminated the heightdifferences as modeled in spite of the addition of height restorationterm in the extended TCA. Larger Δh tended to result in larger d′ asexpected, but the effect of ar was not clear in the d′ plot alone.

Table 2 shows the average height differences of the probe tip,Δpz, across the height change interval measured in the experiment.A few important observations can be made with the table. First,the effect of height compensation was diminished with the increaseof ar, as expected. In the cells under P1→ P2, it is clear that Δpzvalues decreased from a positive number with the increase of ar.When Δh = 0.6 and 0.8 mm for ar = 0.30, the corresponding Δpzvalues turned negative, indicating that a slight height inversion oc-curred similarly to those in Figure 2. In the cells under P2 → P1,Δpz values increased from a negative number with the increase ofar. Four cases exhibited the height inversion, in Δh = 0.6 mm forar = 0.10 and in all Δh values for ar = 0.30. These results are some-what inconsistent with the large positive d′ values measured in allexperimental conditions. The reason was unveiled during subjectdebriefing after the experiment. Most subjects answered that it wasquite difficult to detect a height decrease under the conditions ofP2→ P1 with ar = 0.30. Thus, their strategy was to focus whether

199

a surface had a height increase, which was much easier to feel dueto the lateral contact cue created by surface normal rendering whenthe stylus hit the height-changing interval. The subjects then judgedplanes without such contact sensations to have a decreasing heightchange. It is possible that d′ would have been close to 0 if the taskhad been to discriminate a flat surface (i.e., Δh = 0) from one witha decreasing height change. Second, the height compensation wasconsistently more effective under the P1→ P2 conditions (strokingto a higher plane) than P2 → P1 (stroking to a lower plane) in theΔpz values. This has been unanimously observed in our investi-gations including [3][4] and seems to be related to a more subtleexploratory pattern of the user.

In summary, the results of Experiment III and our testing withvarious parameters have indicated that ar = 0.10 is a good candi-date for the extended TCA. Recall that this value corresponds to thesmallest detection threshold of surface slope perception.

6 CONCLUSIONS AND FUTURE WORK

In this article, we have presented the extended topography compen-sation algorithm that can be used for haptizing the scientific dataof surface topography and stiffness, both continuously varying, ina perceptually correct manner. Our algorithm is based on the hu-man exploratory behavior of force constancy that allows estimatingthe amount of necessary height compensation based on the user-applied force and stiffness difference. A way to cancel out errorsin the user-applied force estimation is also integrated based on thehuman sensitivity to surface slope perception. The performance ofthe extended TCA was experimentally validated through the prox-imal stimuli and the height discriminability. We also reported thedetection threshold of surface slopes rendered via a force-feedbackinterface which can be used for other perception-based haptics ap-plications such as watermarking. In another perspective, the TCAcan be viewed as a variation of impedance rendering techniqueswith a flavor of admittance rendering, attempting to mildly controlthe probe position by estimating user-applied forces based on theforce constancy. The extended TCA brings one-step advance fromthe current practice of haptic rendering which requires constant sur-face stiffness for an adequate delivery of surface shape.

Our future work will be pursued in two directions. First, it wasconsistently observed that the effectiveness of height compensationin the TCA depends on the height change direction, and we plan toinvestigate reasons for this. Second, it may be possible to achievesimilar performance by modulating the stiffness model instead ofthe height model to control the probe-tip height (see Equation 2again). Although this approach is less intuitive, it might remove theneed of defining additional points such as the CSCP. Developing acomputational algorithm for this approach and proving its percep-tual performance are another line of work for the near future.

ACKNOWLEDGEMENTS

This work was supported in parts by a grant No. R01-2006-000-10808-0 from the KOSEF and by the BK 21 program.

REFERENCES

[1] R. S. Avila and L. M. Sobierajski. A haptic interaction method forvolume visualization. In Proceedings of the ACM Conference on Vi-sualization, pages 197–204, 1996.

[2] F. P. Brooks, M. Ouh-Young, J. J. Batter, and P. J. Kilpatrick. ProjectGROPE – haptic displays for scientific visualization. In Proceedingsof the ACM SIGGRAPH Conference, pages 177–185, 1990.

[3] J. Cheon and S. Choi. Perceptualizing a “haptic edge” with varyingstiffness based on force constancy. Lecture Notes on Computer Sci-ence (ICAT 2006), 4282:392–405, 2006.

[4] J. Cheon and S. Choi. Haptizing a surface height change with varyingstiffness based on force constancy: Effect of surface normal rendering.In Proceedings of the World Haptics Conference, pages 84–89, 2007.

[5] V. S. Chib, J. L. Patton, K. M. Lynch, and F. A. Mussa-Ivaldi. Hapticidentification of surfaces as fields of force. Journal of Neurophysiol-ogy, 95(2):1068–1077, 2006.

[6] S. Choi and H. Z. Tan. Perceived instability of virtual haptic texture. I.Experimental studies. Presence: Teleoperators and Virtual Environ-ment, 13(4):395 – 415, 2004.

[7] S. Choi, L. Walker, H. Z. Tan, S. Crittenden, and R. Reifenberger.Force constancy and its effect on haptic perception of virtual surfaces.ACM Transactions on Applied Perception, 2(2), 2005.

[8] G. A. Gescheider. Psychophysics: The Fundamentals. Lawrence Erl-baum Associate, Mahwah, NJ, USA, 3rd edition, 1997.

[9] D. Y. P. Henriques and J. F. Soechting. Bias and sensitivity in the hap-tic perception of geometry. Experimental Brain Research, 150(1):95–108, 2003.

[10] C. Ho, C. Basdogan, and M. A. Srinivasan. Efficient point-based ren-dering techniques for haptic display of virtual objects. Presence: Tele-operators and Virtual Environment, 8(5):477–491, 1999.

[11] M. Ikits, J. D. Brederson, C. D. Hansen, and C. R. Johnson. Acontraint-based technique for haptic volume exploration. In Proceed-ings of the IEEE Visualization Conference, pages 263–269, 2003.

[12] H. Iwata and H. Noma. Volume haptization. In Proceedings of theIEEE Virtual Reality Annual International Symposium, pages 16–23,1993.

[13] S. J. Lederman and R. L. Klatzky. Hand movement: A window intohaptic object recognition. Cognitive Psychology, 19:342–368, 1987.

[14] H. Levitt. Transformed up-down methods in psychoacoustics. Journalof the Acoustical Society of America, 49(2):467–477, 1971.

[15] S. Louw, A. M. L. Kappers, and J. J. Koenderink. Active haptic de-tection and discrimination of shape. Perception & Psychophysics,64(7):1108–1119, 2002.

[16] K. Lundin, A. Ynnerman, and B. Gudmundsson. Proxy-based hapticfeedback from volumetric density data. In Proceedings of the Euro-haptics, pages 104–109, 2002.

[17] R. Maciejewski, S. Choi, D. S. Ebert, and H. Z. Tan. Multi-modalperceptualization of volumetric data and its applicatiton to moleculardocking. In Proceedings of the World Haptics Conference, pages 511–514, 2005.

[18] M. S. O’Modhrain. Restricted access: Exploratory procedures andobject properties. In Proceedings of the ASME Dynamic Systems andControl Division, volume 67, pages 415–420, 1999.

[19] D. Prattichizzo, M. Barni, G. Menegaz, A. Formaglio, H. Z. Tan, andS. Choi. Perceptual issues in haptic digital watermarking. IEEE Mul-timedia, 14(3):84–91, 2007.

[20] D. Ruspini, K. Kolarov, and O. Khatib. The haptic display of complexgraphical environments. In Proceedings of ACM SIGGRAPH, pages345–352, 1997.

[21] K. Salisbury and C. Tarr. Haptic rendering of surfaces defined byimplicit functions. In Proceedings of the ASME Dynamic Systems andControl Divisioin, volume 61, pages 61–67, 1997.

[22] D. Sarid. Scanning Force Microscopy. Oxford Unversity Press, NewYork, 1991.

[23] R. M. Taylor II, W. Robinett, V. L. Chi, F. P. Brooks, Jr., W. V. Wright,R. S. Williams, and E. J. Snyder. The nanomanipulator: A virtual-reality interface for a scanning tunneling microscope. In Proceedingsof the ACM SIGGRAPH Conference, pages 127–134, 1993.

[24] B. J. van der Horst and A. M. L. Kappers. Curvature discrimination invarious finger conditions. Experimental Brain Research, 177(3):304–311, 2007.

[25] J. Voisin, G. Benoit, and E. C. Chapman. Haptic discrimination ofobject shape in humans: Two-dimensional angle discrimination. Ex-perimental Brain Research, 145(2):239–250, 2002.

[26] H. Yano, M. Nudejima, and H. Iwata. Development of haptic render-ing methods of rigidity distribution for tool-handling type haptic inter-face. In Proceedings of the World Haptics Conference, pages 569–570,2005.

[27] C. B. Zilles and J. K. Salisbury. A constraint-based god-object methodfor haptic display. In Proceedings of IEEE International Conferenceon Intelligent Robots and Systems, pages 146–151, 1995.

200