Discrimination of Virtual EnvironmentsUnder Visual and Haptic Rendering Delays

In Lee and Seungmoon ChoiHaptics and Virtual Reality Laboratory

Department of Computer Science and EngineeringPOSTECH, Republic of Korea{inism, choism}@postech.ac.kr

Abstract

Many virtual reality systems have a distributed structurefor certain purposes such as more computational power,tele-presence, collaboration, and portability. However, net-work delays are inevitable in the distributed structure andoften make sensory information delivered behind time to theuser. In the literature, the effect of network delays on thequality of virtual environments has been considered mostlywith respect to task performances. In this paper, we pay at-tention to whether perceptual artifacts caused by networkdelays are perceptible by the user, which is a more stringentcriterion than the degradation of task performance. We ex-amined minimum perceptible visual and/or haptic render-ing delays by measuring their discrimination thresholds be-tween normal and delayed virtual environments with andwithout a task, and report the results in this paper. We alsoprovide a simple guideline for determining whether someactive delay compensation algorithms are required in a dis-tributed virtual reality system by comparing representativenetwork delays to the measured discrimination thresholds.

1 Introduction

This paper presents an experimental study that investi-gated the effects of network delay in visual and haptic ren-dering on the user’s perception of virtual environments. Forvarious purposes, a virtual environment and related render-ing can be managed in separate computers, which causesinevitable network delays. One such example is a collabo-rative virtual environment where multiple remotely-locatedusers share, explore, and interact within a same virtual envi-ronment [8][6]. Virtual environment models shared in mul-tiple computers need to be synchronized, but it is often hin-dered by network delays. On the other hand, the distributed

visual and haptic rendering routines can be used in order toimprove the overall computational performance and/or theindividual reusability of visual and haptic devices [5].

Depending on the configuration of a networked virtualenvironment, the sensory information that a user perceivesmay be delayed from a virtual environment model visuallyand/or haptically. Such rendering delays can induce variousperceptual artifacts such as delayed pointer movements ona screen and abruptly changing forces. Therefore, appropri-ate design of networked virtual reality applications requiresproper understanding of the amount of tolerable delay andits effect on perception and/or tasks. It is then possible todetermine whether to apply various methods developed tocope with network delays in a distributed system (see [3][4]for a survey).

In fact, there have been several studies for this issue, butmost of them were concerned with the effect of network de-lays on task performances. As an early study without forcefeedback, MacKenzie and Ware showed that visual delaycan increase task completion time and error rate in a 1Dtarget acquisition task using a mouse, especially when vi-sual delay is above 75 ms [11]. A similar experiment wasconducted including haptic feedback by Jay and Hubbold[7]. With a task to tap two targets reciprocally using a hap-tic device, it was reported that a visual delay larger than 50ms significantly increases movement time and error rate andthat a haptic delay larger than 150 ms significantly degradesmovement time.

In this paper, we investigate the effect of network de-lay on virtual environments using a more fundamental andstringent criterion, that is, whether or not a user can per-ceive perceptual artifacts occurred due to the delay. Suchartifacts greatly undermine the realism of virtual environ-ments, and thus should be considered as a crucial measurefor dealing with network delays. Using two virtual envi-ronments (with and without rendering delays), the degreeof rendering delays that resulted in noticeably different vir-

Frontiers in the Convergence of Bioscience and Information Technologies 2007

0-7695-2999-2/07 $25.00 © 2007 IEEEDOI 10.1109/FBIT.2007.124

560

Frontiers in the Convergence of Bioscience and Information Technologies 2007

0-7695-2999-2/07 $25.00 © 2007 IEEEDOI 10.1109/FBIT.2007.124

560

Frontiers in the Convergence of Bioscience and Information Technologies 2007

0-7695-2999-2/07 $25.00 © 2007 IEEEDOI 10.1109/FBIT.2007.124

560

Frontiers in the Convergence of Bioscience and Information Technologies 2007

0-7695-2999-2/07 $25.00 © 2007 IEEEDOI 10.1109/FBIT.2007.124

554

Figure 1. A subject performing the experi-ment.

tual environments was estimated in a psychophysical ex-periment. The experiment examined three cases where ren-dering delays were present in the visual channel only, thehaptic channel only, and both visual and haptic channels,respectively. The effect of imposing a task on the user wasalso considered. The smallest rendering delays needed todiscriminate normal and delayed virtual environments weremeasured using the adaptive staircase procedure [10]. Fi-nally, we compared the measured discrimination thresholdswith representative network delays, which can be useful todecide whether to employ active delay compensation in net-worked virtual reality applications.

The rest of this paper is organized as follows. In Section2, the detailed methods used to design the experiment areprovided. Sections 3 and 4 describe the experimental resultsand discussions on the results, respectively. We concludethis paper in Section 5.

2 Methods

This section describes the methods used to design andconduct the psychophysical experiment.

2.1 Apparatus

A haptic interface used in the experiment was a PHAN-ToM 1.0A (Sensable Inc., USA) shown in Figure 1. Its per-formance has been rigorously characterized in the literature[1][2] and is generally regarded to be adequate for hapticpsychophysical studies. Visual information about a virtualenvironment was provided through a 19-inch LCD monitor(Dell Inc.).

Figure 2. Virtual environment used in the ex-periment.

2.2 Subjects

Six healthy university students participated in the exper-iment and were paid for their service. Their ages variedin 23 – 28 years with an average of 24.3. Three subjectswere female (subjects S1, S5, and S6), and the others weremale. All of the subjects were right-handed by self-reportand naive, i.e., had not been exposed to any haptic inter-faces prior to their participation to the present experiment.No subjects reported any known sensory abnormalities ontheir upper extremities.

2.3 Stimuli

The virtual environment that the subjects explored wasconsisted of three elements as shown in Figure 2. A smallcone represented the position of the haptic interaction point(HIP; the end point of the PHANToM stylus). A blueopaque cube was the object that could be grasped andmoved by the user. A red transparent cube showed the des-tination where the blue cube needed to be moved by thesubject. The two cubes were of equal size, and their edgelength was about 2.2 cm on the monitor and 1 cm in theworkspace of the PHANToM device. The boundary of thevirtual environment was also cubic with the edge length of20 cm in vision and 9 cm in touch. The haptic renderingused the virtual proxy algorithm [12] and the stiffness ofthe blue cube was set to 0.3 N/mm.

In the virtual environment, the subject could touch theboundaries of the virtual environment or the blue cube bymaking contacts with them using the PHANToM stylus.The subject could also drag the blue cube by touching it andpressing a button on the stylus. During dragging, a resistive

561561561555

Delay TypeVisual Only Haptic Only Both Visual and Haptic

Task TypeWithout Task T0Dv T0Dh T0Dvh

With Task T1Dv T1Dh T1Dvh

Table 1. Experimental conditions.

force simulating viscosity friction was rendered to providethe subject with a sensory cue for dragging. The blue conefor the HIP was turned to red while it was in contact withany objects. Visual and haptic update rates were set to 60Hz and 1 kHz, respectively.

2.4 Experimental Conditions

Two independent variables were defined for the exper-iment. One independent variable was the type of render-ing delay: Dv – visual delay only, Dh – haptic delay only,and Dvh – both delays. For each delay type, the render-ing of corresponding modality (or modalities) was delayedwith respect to the current information of the virtual envi-ronment. The other independent variable was related to theexistence of a task: T0 – no task and T1 – with task. In T0,the subject was instructed to freely explore the virtual envi-ronment. In T1, the subject was required to complete a task,that is, moving the blue cube to a destination represented bythe red cube as fast as possible. It follows that there weretotal six experimental conditions (three delay types and twotask types) as summarized in Table 1 .

For cost efficiency, the experiment was designed follow-ing the within-subject design. To avoid any order effects,the order of experimental conditions was balanced acrossthe subjects using the Latin Square method [13].

2.5 Procedures

For each experimental condition, a discriminationthreshold between the normal and delayed virtual environ-ments was estimated using the two-interval, forced-choice,one-up three-down staircase adaptive method [10][9]. Inthis method, each trial consisted of two intervals. One in-terval was the reference that contained no additional delay,whereas the other was a test interval that included a delayin the rendering of the corresponding modalities. The pre-sentation order of the two intervals was randomly chosen ineach trial. Each interval lasted for 5 seconds, and the inter-stimulus time between the intervals was set to 1 second.

Before each trial, the initial position of the blue cube andthe destination position were randomly chosen in the virtualenvironment and were used for both intervals. A cautionwas taken to prevent the blue cube from being very closeto the destination, such that the distance between them was

always larger than the half of their size. The subject was in-structed to interact with the virtual environment either freely(under T0) or performing a task (under T1). In particular,in the experimental conditions under T1, the subject wasasked to move the blue cube to a destination represented bythe red cube as fast as possible. When the subject touchedthe blue cube, the blue cone representing the HIP turned tored. While the subject pressed the button on the PHANToMstylus, the blue cube became attached to the stylus and thesubject could control the position of the blue cube using thestylus. The subject would then move the cube to the des-tination and release the button. If their distance was closerthan the half of the cube edge length, the cube was auto-matically aligned to the destination and the cube color waschanged to red. An additional haptic cue (a clicking force)was also generated as a signal for task completion. Afterthis, the blue cube either stayed in the destination (underT0) or was relocated to the initial position immediately forrepeated task conduction (under T1). The subject was alsoinformed that the task could be repeated during the allowedtime of an interval and the number of task completion wascounted for task performance. This was to keep the subjectconcentrating on the task, which could lead to a significantcognitive load against perceiving the artifacts due to render-ing delays. At the end of each trial, the subject was asked toanswer which interval was unnatural, by pressing key ‘1’ ifthe first interval seemed unnatural, or key ‘2’ if the secondinterval seemed unnatural. A next trial started immediatelyafter the subject responded to the question.

In each experimental condition, the first trial began witha large delay (= 200 ms). This amount of delay was largeenough to make the delayed virtual environment contain ob-vious artifacts, enabling perfect discrimination of the refer-ence and test intervals. In the beginning, the delay was de-creased for every correct response with a large step size forfast threshold convergence. After the first response rever-sal, three consecutive correct responses were required forthe decrease of the delay. On the other hand, one incorrectanswer led to an increase of the delay. This procedure al-lows efficient estimation of a discrimination threshold cor-responding to the 79.4 % percentile point on a psychometricfunction. The step size was 60 ms in the first trial, and wasreduced as step size = 60 ms / n, where n is the trial number,until it reached the minimum step size (= 2 ms). A sessionwas terminated after 14 reversals (a case where an increas-

562562562556

dropped

Figure 3. Sample results of an experimentalcondition (subject S5, T0Dv).

ing delay sequence was changed to a decreasing one, andvice versa). Most sessions consisted of 70 – 100 trials.

Before beginning the experiment, each subject was pro-vided with detailed explanations about the experimentalprocedure by the experimenter, and also had a short trainingsession. Each experimental condition took about 20 min-utes to complete. The subject was required to take a rest forfive minutes before beginning a next experimental condi-tion. The whole experiment took about two and half hours.

2.6 Data Analysis

The method used to compute discrimination thresholdsare described with the aid of Figure 3 that represents theresults of subject S5 under the experimental condition ofT0Dv. The figure shows delay values used in each trial. Ineach experimental condition, the delay values at which 14response reversals occurred (e.g., the red circles and greensquares in Figure 3) were recorded. The delay values at thefirst two reversals (e.g., the red circles in Figure 3) were dis-carded due to the large step sizes. Among the last 12 delayvalues (e.g., the green squares Figure 3), each two consecu-tive delay values were paired and averaged, resulting in sixestimates for a discrimination threshold. The six thresholdestimates were used in statistical analysis, and their meanwas considered as the discrimination threshold of the ex-perimental condition.

3 Results

The measured discrimination thresholds of each subjectare provided in Table 2. The discrimination thresholds av-eraged across the subjects are also shown in Figure 4 alongwith the error bars representing the standard errors. Over-all, the average discrimination thresholds varied in 11.83 –

Dv (ms) Dh (ms) Dvh (ms)

T0

S1 59.33 3.50 5.50S2 63.25 4.33 6.58S3 50.33 16.83 12.42S4 84.50 20.75 15.25S5 19.17 17.00 15.75S6 41.67 8.58 19.83

Avg. 53.04 11.83 12.56

T1

S1 138.25 21.92 31.75S2 125.25 85.92 71.75S3 46.67 45.17 34.00S4 62.17 76.33 32.33S5 35.00 14.67 14.25S6 95.42 20.58 23.42

Avg. 83.79 44.10 34.58

Table 2. Measured discrimination thresholds.

Figure 4. Average discrimination thresholds.

83.79 ms.Two-way ANOVA showed that both delay type (Dv, Dh,

and Dvh) and task type (T0 and T1) were statistically sig-nificant in determining the discrimination thresholds at sig-nificance level 0.05 (F2,10 = 14.71 and p < 0.001 for delaytype; F1,5 = 8.65 and p = 0.0322 for task type), but theirinteraction was not (F2,10 = 0.25 and p = 0.7823). In a pair-wise comparison test using the SNK test, all delay types be-longed to statistically different groups at significance level0.05. Conditions with visual delay produced the largest av-erage discrimination thresholds, and those with both visualand haptic delays resulted in the smallest. In subject de-briefing after the experiment, all subjects reported that theynoticed that the pointer movement displayed on the screencan be delayed from the stylus movement controlled by the

563563563557

subjects and used it as a sensory cue to declare an unnaturalvirtual environment in the conditions with visual delay. Inthe conditions with haptic delay, the subjects reported thatthey sometimes felt excessively large abrupt forces whenthey initiated contacts with the blue cube or the virtual en-vironment boundary and used this artifact as a primary cue.Our data indicate that the haptic artifact can be much moreevident with much smaller delays than the visual artifact.In terms of the task type, the subjects were less sensitive toperceptual artifacts due to rendering delays when they wererequired to perform a specific task as expected.

4 Discussion

The measured discrimination thresholds can be com-pared to previously published delay values that significantlyaffected task performances. In [7], it was shown that visualdelays larger than 50 ms significantly degraded task com-pletion time. In our experiment, the average discriminationthreshold (= 53.04 ms) under the visual delay conditionswithout the task (T0Dv) was comparable to 50 ms, but that(= 83.79 ms) with the task was (T1Dv) much larger than50 ms. This seems to be due to the fact that our task wasmore difficult than the 1D tapping task used in [7]. The de-lay value that caused clear performance drop under hapticdelay was about 150 ms in [7]. In this case, our thresholdswere much smaller, suggesting that fairly large haptic de-lays that are clearly perceivable can still allow reasonabletask performance.

Typical network packet travel times measured for sev-eral representative places are provided in Table 3. For themeasurement, the servers listed in the table were pingedwith a 32 byte packet 100 times. Packets that were notreturned in 1 second were dropped. Each round-trip time(RTT) was calculated by averaging the ping test results ofreturned packets, and one-way travel time (RTT/2) was setto the half of the round-trip time. Note that depending ona network configuration, either packet round-trip time orone-way travel time can be used as an estimate of render-ing delay. For example, in a collaborative virtual environ-ment with the client-server architecture [8][6], the input ofa user must be sent to the server for synchronization, andvisual and haptic rendering can be processed in the clientafter receiving updated results from the server. In this case,the round-trip time can be an adequate estimate for render-ing delay. On the other hand, in a system where visual andhaptic rendering are performed in separate computers [5],the input of a user is transferred from one computer to theother computer that can display the new information imme-diately after receiving it. The one-way travel time is a moreappropriate measure in this scenario.

By comparing the packet travel times and the discrimi-nation thresholds shown in Figure 4, we can appreciate the

needs for the active compensation of network delays in therendering of a virtual environment. For example, visual andhaptic delays between computers within the same networkresulted in very small packet round-trip time (< 1 ms; mea-sured within the POSTECH campus), which is not to beperceived by the virtual environment users. The round-triptime between computers in Seoul, Korea and the POSTECH(about 270 km apart) was 12 ms on average, slightly overthe haptic discrimination thresholds under the no task con-dition. The same test conducted between computers inJapan (yahoo.co.jp) and the POSTECH yielded only44 successfully received packets. The rest were droppedbecause they could not be returned within 1 second afterthey were sent, which is another major problem in network-based virtual environments. The average round-trip timewas 57 ms, which causes clearly perceivable artifacts due tohaptic delays and makes barely discernable virtual environ-ments due to visual delays. An experiment with a computerin USA (yahoo.com) showed 32 dropped packets and 207ms average round trip time, indicating apparent needs forcompensation.

Such comparisons can be made more easily using Figure5. In the figure, the orange solid bars represent the aver-age packet round-trip times, and the green shaded bars theone-way travel times. The bars are grouped by the distancebetween different locations. The discrimination thresholdsmeasured in the present study are indicated by the hori-zontal lines. Note that the thresholds of T0Dh and T0Dvhconditions are so close to be overlapped in one line. If abar (network delay) crosses a horizontal line (discrimina-tion threshold), it means the user would suffer from percep-tual artifacts due to the delay in the corresponding config-uration (determined by the existence of a task, the delayedsensory channel, and the type of packet travel times). Notethat in the comparisons other important factors such as net-work jitter and packet loss are ignored. Including them intoconsideration is likely to further increase the need for activedelay compensation in distributed virtual environments.

5 Conclusions

We reported a psychophysical experiment that investi-gated minimum delay values required for the users to reli-ably discriminate normal and delayed virtual environments.Delays for visual, haptic, and both visual and haptic sen-sory channels were considered with and without a task.The measured discrimination thresholds were shown to besmaller than the previously reported delay values warrant-ing good task performances, and could be problematic fornetworked virtual environments between different coun-tries. This imposes a more stringent constraint on devel-oping convincing networked virtual environments than thepreviously reported studies.

564564564558

Returned Average Round Average One-wayLocation (Server URL) Packets Trip Time (ms) Travel Time (ms)

POSTECH (postech.ac.kr) 100 < 1 < 1Seoul, Korea (kr.yahoo.com) 100 12 (5 – 23) 6 (2.5 – 11.5)

Japan (yahoo.co.jp) 44 57 (54 – 62) 23.5 (27 – 31)USA (yahoo.com) 68 207 (184 – 225) 103.5 (92 - 112.5)

Table 3. Example network packet travel times. 100 packets were sent in total for the test with eachlocation. The numbers after the average times represent the range of measured times.

Figure 5. Comparison between the averagenetwork packet travel times and the delayperception thresholds.

Acknowledgment

This work was supported in parts by a grant No. R01-2006-000-10808-0 from the Korea Science and Engineer-ing Foundation (KOSEF) funded by the Korea govern-ment(MOST) and by the Brain Korea 21 program from theKorea Research Foundation.

References

[1] M. C. Cavusoglu, D. Feygin, and F. Tendick. A criti-cal study of the mechanical and electrical properties of thePHANToMT M haptic interface and improvements for highperformance control. Presence, 11(6):555–568, 2002.

[2] S. Choi and H. Z. Tan. Perceived instability of virtual haptictexture. I. Experimental studies. Presence: Teleoperatorsand Virtual Environment, 13(4):395 – 415, 2004.

[3] D. Delaney, T. Ward, and S. McLoone. On consistency andnetwork latency in distributed interactive applications: Asurvey – part I. Presence: Teleoperators and Virtual En-vironments, 15(2):218–234, 2006.

[4] D. Delaney, T. Ward, and S. McLoone. On consistency andnetwork latency in distributed interactive applications: Asurvey – part II. Presence: Teleoperators and Virtual En-vironments, 15(4):465–482, 2006.

[5] E. Dorjgotov, S. Choi, S. R. Dunlop, and G. R. Bertoline.Portable haptic display for large immersive virtual environ-ments. In Proceedings of the 14th Symposium on Haptic In-terfaces for Virtual Environment and Teleoperator Systems(HAPTICS 06), pp. 321–327, 2006.

[6] C. Gunn, M. Hutchins, and M. Adcock. Combating latencyin haptic collaborative virtual environments. Presence: Tele-operators and Virtual Environments, 14(3):313–328, 2005.

[7] C. Jay and R. Hubbold. Delayed visual and haptic feedbackin a reciprocal tapping task. In Proceedings of the WorldHaptics Conference, pp. 655–656, 2005.

[8] J. Kim, H. Kim, B. K. Tay, M. Muniyandi, M. A. Srinivasan,J. Jordan, J. Mortensen, M. Oliveira, and M. Slater. Transat-lantic touch: A study of haptic collaboration over long dis-tance. Presence: Teleoperators and Virtual Environments,13(3):328–337, 2004.

[9] M. R. Leek. Adaptive procedure in psychophysical research.Perception & Psychophysics, 63(8):1279–1292, 2001.

[10] H. Levitt. Transformed up-down methods in psychoa-coustics. Journal of the Acoustical Society of America,49(2):467–477, 1971.

[11] I. S. MacKenzie and C. Ware. Lag as a determinant of hu-man performance in interactive systems. In Proceedings ofthe SIGCHI Conference on Human Factors in ComputingSystems, pp. 488–493, 1993.

[12] D. Ruspini, K. Kolarov, and O. Khatib. The haptic displayof complex graphical environments. In Computer GraphicsProceedings (ACM SIGGRAPH Proceedings), pp. 345–352,1997.

[13] B. J. Winer, D. R. Brown, and K. M. Michels. StatisticalPrinciples in Experimental Design. McGraw-Hill, 3rd edi-tion, 1991.

565565565559