Control gain adaptation in virtual reality mediated human–telerobot interaction

Control Gain Adaptation in Virtual Reality MediatedHuman–Telerobot Interaction

Mohamed A. Sheik-Nainar and David B. KaberDepartment of Industrial Engineering, North Carolina State University,Raleigh, North Carolina

Mo-Yuen ChowDepartment of Electrical and Computer Engineering,North Carolina State University, Raleigh, North Carolina

ABSTRACT

The Internet connects millions of computers worldwide, and provides a new potential workingenvironment for remote-controlled telerobotic systems. The main limitation of using the Internet inthis application is random delays between communicating nodes, which can cause disturbancesin human–machine interaction and affect telepresence experiences. This is particularly importantin systems integrating virtual reality technology to present interfaces. Telepresence, or the sense ofpresence in a remote environment, hypothetically is positively related to teleoperation task perfor-mance. This research evaluated the effect of constant and random network (communication) delayson remote-controlled telerover performance, operator workload, and telepresence experiences. Theresearch also assessed the effect of using a system gain adaptation algorithm to offset the negativeimpact of communication delays on the various response measures. It was expected that with gainadaptation, system stability, performance, and user telepresence experiences would improve with acorresponding decrease in workload. Results indicated that gain adaptation had a significant effecton the performance measures. The study demonstrated that gain adaptation could reduce deterio-ration in telepresence experiences and improve user performance in teleoperated and teleroboticcontrol. © 2005 Wiley Periodicals, Inc.

1. INTRODUCTION

The Internet, because of its affordability, accessibility, extensive applications, and well-developed infrastructure, has been investigated as an alternative for remotely controllingreal-time systems, such as robotic systems for tele-manufacturing, tele-training, tele-services, etc. A number of prototypes of Internet-based telerobotic systems were devel-oped following the advent of the on-line Cambridge Coffee Pot in 1994 (Taylor &Trevelyan, 1995). Hu, Yu, Tsui, and Zhou (2001) classified this and other teleoperationsystems (on-line at that time) as first generation Internet-based telerobots based on theirrequirement for direct human control and minimal on-board (remote) automation. Future

Correspondence to: David B. Kaber, 2401 Stinson Drive, 328, Riddick Labs, Box 7906, Raleigh, NC 27695–7906. E-mail: [email protected]

Human Factors and Ergonomics in Manufacturing, Vol. 15 (3) 259–274 (2005)© 2005 Wiley Periodicals, Inc.Published online in Wiley InterScience (www.interscience.wiley.com). DOI: 10.1002/hfm.20025

259

Internet-based tele-manipulation systems are expected to integrate human control withmore remote system autonomy to promote overall teleoperation performance and usercontrol satisfaction (Hu et al., 2001).

Internet-based control systems must rely on available communication protocols toexchange real-time data between local and remote sites. The most commonly used pro-tocols are the Transmission Control Protocol (TCP) and User Datagram Protocol (UDP)(see the Appendix for a definition of expert acronyms used throughout this article). Bothof these protocols have unique characteristics, which are not conducive to the require-ments of real-time systems. Thus, the Internet can be described as a bandwidth limited,unreliable communication medium with random delay. Because of these shortcomings,Hu et al. (2001) suggested that the next generation of Internet-based systems must alsohave a high degree of tolerance to possible data packet losses and include innovativemechanisms for coping with shared user control. Conventional teleoperation systems oftenintegrate video cameras for visual feedback, which require high communication band-widths. This is not possible in the case of an Internet-based system because of the limitedbandwidth availability and lack of guaranteed reliability. Therefore, sophisticated inter-faces have become increasingly important for Internet-based teleoperation, particularlypredictive displays, in an attempt to account for communication delay issues and promoteaccurate operator perception of system states (Bejczy, Kim, & Venema, 1990; Eberst,Stoffler, Barth, & Farber, 1999).

Virtual environments (VE) or virtual reality (VR) technology has been used in recentresearch to create intuitive, multimodal, interfaces to teleoperate systems by taking advan-tage of the “bandwidth” of several human sensory channels simultaneously to a greaterextent than is possible with conventional display technologies (e.g., Hine, Hontalas, Fong,Piguet, Nygren, & Kline, 1995; Kaber, Riley, Zhou, & Draper, 2000). Such interfaces canbe effectively developed as predictive control displays or off-line programming tools toaccount for communication problems through use of highly realistic graphical models ofremote manipulators and system dynamics models.

Contemporary VR-based interfaces for teleoperation systems have been designed withthe objective of providing users with the sense of being part of a realistic environment, ordirect (hands-on) performance of tasks. This sensation has been labeled telepresence andis considered a design ideal for VR interfaces (Draper, Kaber, & Usher, 1998). It has beenhypothesized by many researchers that telepresence shares a positive relationship withvirtual task performance or teleoperation (Sheridan, 1992) and, consequently VR andteleoperator design for telepresence has been advocated. Nash, Edwards, Thompson, andBarfield (2000) classified the factors affecting telepresence as computer related, commu-nication related, individual related, task related, and external disturbances. The currentresearch focused on the effects of disturbances from the surrounding environment ontelepresence, performance, and workload in Internet-based teleoperation. Nash et al. saidthat a user’s sense of immersion and presence in a VE might be degraded when real-worldstimuli are present about VE displays. These stimuli may serve as attentional distracters;thus, less attention is focused on the VE (Draper et al., 1998). Network communicationdelays may be considered as one such disturbance (Nash et al., 2000) affecting telepres-ence. Even though VR is seen as one potential solution to communication delay problemsin teleoperation, it may still have limitations because of degradations in telepresence. Inthis study, we examined the effects of different types of communication network delays—no-delay, constant (dedicated network) delay, and random (Internet) delay on operatorperformance, workload and telepresence experiences in a telerover navigation operation

260 SHEIK-NAINAR, KABER, AND CHOW

involving the use of a VR interface, in part, to assess the utility of the interface undervarious remote control conditions.

Control systems research has been conducted to address delay-in-the-loop issues forsome time now. For example, during the 1960s, significant work was devoted towardsautomated compensation for constant time delays in satellite control systems. However,since the late 1990s, major efforts have been focused on addressing time-varying, sto-chastic Internet delays in the control loop of a highly nonlinear system (Tipsuwan &Chow, 2003a), especially to reduce the impact of communication delay on system stabil-ity and safety. In general, overall teleoperation system performance depends on the timeconstant of the system itself, as well as the time delay caused by network control com-munication. For example, a large robot arm with a time constant of seconds will generallybe insensitive to a time delay of tens-of-milliseconds. While the performance of a micro-machine with time constant of milliseconds will be highly sensitive to network-induceddelays of milliseconds. With respect to contemporary control system research on time-varying delays, Luck and Ray (1990) proposed a system-state predictor using memorybuffers to convert random network delays into time-invariant delays. Nilsson, Bernhards-son, and Wittenmark (1990) utilized an optimal stochastic control concept by treatingnetwork delay effects as a Linear Quadratic Gaussian (LQG) problem. Walsh, Ye, andBushnell (1999) used nonlinear control theory to formulate network delays as a vanishingperturbation. Tipsuwan and Chow (2001) proposed a real-time system gain adaptationapplication to compensate for network quality of service (QoS) variation and deteriora-tion in a teleoperation scenario. When there is a change in the network delay or through-put, the adaptation scheme automatically adjusts the gain of the teleoperation systemcontroller to reduce the impact of the network delay (Tipsuwan & Chow, 2003b) on per-formance, for example navigation errors and collisions.

The present research also evaluated the utility of a gain adaptation approach, similar toTipsuwan and Chow’s (2001) approach, to compensate for negative impacts of delay ontelepresence, performance, and operator workload in a basic teleoperation task. In gen-eral, we hypothesized that with gain adaptation for various types of network delays, tele-operation system stability, performance, and user telepresence experiences with the VRinterface would improve with a corresponding decrease in workload.

2. METHOD

2.1. Teleoperation System

In this study, we simulated a telerover performing a simple obstacle navigation task. Therover was modeled based on a differential drive mobile robot with two driving wheelsand one caster wheel, and was controlled over a simulated network. Figure 1 shows theschematic of the simulated system. Two levels of control/automation (LOA) were imple-mented in the simulation including teleoperated and telerobotic control. In the teleoper-ation mode, the user directly controlled the telerover, while in the telerobotic mode boththe user and machine shared control.

2.1.1. Main Controller. The main controller, as part of the simulation, computed con-trol signals for the local controller and tracked the path of the telerover. In the teleoper-ation mode of control, the controller computed the angular velocity of the individual rover

HUMAN-TELEROBOT INTERACTION 261

wheels based on the control input commanded directly from the user. In the case of thetelerobotic control mode, the user specified a target location and the controller computedthe motion path using a quadratic path-tracking algorithm (Tipsuwan & Chow, 2003b;Yoshizawa, Hashimoto, Wada, & Mori, 1996).

2.1.2. Local Controller. The local controller module simulated two Proportional-Integral controllers. Each controller directed the speed of one driving wheel of the rover.The local controller received reference angular velocities for the wheel from the maincontroller.

2.1.3. Network. Network QoS deterioration was simulated in this study by randomlygenerating variable inter-arrival times (or jitter) of data packets transmitted between theoperator control interface and the simulated rover. Deterioration in QoS is primarily affectedby the inter-arrival time between packets, assuming there are no packet losses.

2.1.4. Main Controller Adaptation. For the telerover to track a trajectory accu-rately under a nominal network QoS condition and constraints on performance (i.e., devi-ation from the trajectory to be tracked), three adaptation parameters a, b, and dmax wereestablished. These parameters represented the main controller adaptation, the referenceposition projection, and maximal distance between the robot and reference point. WhenQoS deterioration occurs, the gain of the main controller may not be suitable for thenetwork condition and robot state. Consequently, the telerover could deviate from thereference trajectory to an unacceptable track because of improper speed and projectedreference points. Thus, the telerover gracefully degrades its performance by adapting itselfto maintain its stability as much as possible under the current network QoS.

This adaptation was applied to the main controller of the simulated teleoperation sys-tem. Extensive experiments were conducted to determine an optimal range of values fora, b, and dmax within the delay bounds of 750 ms–250 ms. That is, control gains werepredetermined to facilitate optimal teleoperation performance under various rover oper-ating conditions and network traffic conditions. The gains were then stored in the memory

Figure 1 Schematic of simulated telerover system.


of the main controller in a “look-up” table. Depending on the operating conditions anddelays, the system would appropriately adapt the controller gain setting. Details of thegain adaptation can be found in Tipsuwan (2003) and Tipsuwan and Chow (2004). Pre-vious research has noted that a lag of 225 ms results in performance degradation in amotor-sensory task, and a delay of 1000 ms tremendously impairs performance. The termlag usually refers to the time delay due to system dynamics while the term delay refers tothe actual time delays caused by network communication. In some situations, it may beconvenient to lump the terms lag and delay together for the ease of discussion. Thus, inthe present study, the terms lag and delay are used interchangeably to signify the delay insystem performance subject to either network-induced transportation time delay or lagcaused by system dynamics. The delay values tested here are representative of the worstdelays in previous VR research but are in the practical range for Internet-based teleoper-ation, considering the average communication delay between North America and othercontinents ranges from 116.47 ms to 1257 ms (Fei, Pei, Liu, & Zhang, 1998).

2.1.5. Interface Design. The VR interface for telerover navigation control consistedof four windows (see Figure 2) including: (a) a main window, which displayed an exo-centric view of the telerover; (b) an aerial view of the telerover and its operating envi-ronment; (c) a virtual joystick control panel; and (4) a speedometer display. The mainwindow facilitated 3D viewing of the virtual environment.

The aerial view was integrated to provide a better overall sense of the environment andto facilitate operator judgments of the position of the rover relative to task objects. It alsoprovided features like panning and zooming of the displayed view. Additionally, this win-dow was utilized in the telerobotic control mode to select a destination to which the roverwas to navigate. This was accomplished by moving virtual crosshairs on the display. Thecrosshairs are visible in Figure 2 as a “�” in the aerial window, and directly in front of therover and to the right in the main window (looks like a “spiked hockey puck”).

Figure 2 Virtual reality interface for telerover navigation.


The joystick navigation control was used in the teleoperation control mode for impart-ing motion to the telerover. The control included eight directional arrows correspondingto forward, backward, left, right, forward-left, backward-left, forward-right, and backward-right motion. The speedometer was a passive display with the purpose of giving operatorsa sense of how fast the rover was traveling in the environment.

2.2. Experiment Design

2.2.1. Task. The task as part of the telerover control simulation was to navigate therover in a desert-like VE between obstacles (virtual oil drums), much like a downhillslalom ski race. At the start of the simulation, all obstacles were randomly positionedabout the VE and were colored blue or red. Navigating the telerover between a blue obsta-cle (barrel) and its nearest red neighbor caused the blue obstacle to turn green in color. Asound cue was associated with this event as a redundant indication that an obstacle hadbeen cleared. Colliding with any obstacle caused it to turn black in color (whether it wasoriginally blue, red, or green) and a redundant sound cue was provided. This was con-sidered a performance error and was recorded during the simulation. If a blue obstaclewas involved in a collision causing it to turn to black, then the obstacle could not be madegreen in color. However, in the case of a red obstacle, if it was the nearest neighbor to ablue obstacle, a user was still required to navigate between the blue and black obstacle toclear the blue obstacle (i.e., turn it green). The goal of the task was to convert all blueobstacles in the environment to green. The time required to complete a set of obstaclesand the number of collisions was recorded during simulation trials.

2.2.2. Independent Variables. The independent variables manipulated in this studyincluded: (a) the network delay type, (b) the LOA, and (c) the system gain adaptation tonetwork delays. Three general delay conditions were examined including a no-delay (con-trol) condition, a constant delay representing a teleoperation system using a dedicatednetwork, and a random delay condition to model Internet-based teleoperation. MacKen-zie and Ware (1993) argued that lag has been shown to degrade human performance inmotor-sensory tasks with interactive systems. They found that with a 75 ms lag, an effectcan be easily measured, and at 225 ms, performance is substantially degraded. Watson,Walker, Ribarsky, and Spaulding (1998) claim that a mean delay of 259 ms, with a stan-dard deviation of 83 ms, has a major negative effect on performance. Eberst et al. (1999)said that human operators easily recognize a delay of 250 ms, while a delay of about1000 ms tremendously impairs performance. In this study, a lag of 1000 ms was used forthe constant delay condition, while a lag ranging between 750 ms–1250 ms was usedfor random delay.

Two settings of LOA were used, including direct teleoperation, or operator control, andtelerobotic, or shared human-machine control. Finally, two settings of control gain adap-tation were used including adaptation (ON) and no-adaptation (OFF).

2.2.3. Dependent Variables. The dependent measures of interest in this study includedperformance, telepresence, and workload. The time-to-navigate through the entire set ofobstacles (time-to-task completion [TTC]) and the number of navigation errors/collisionswith obstacles were observed as performance measures and were captured by the com-puter workstation running the simulation.

Telepresence was measured using a two-item Presence Questionnaire (PQ) developedby Draper and Blair (1996). The items in the questionnaire included: (a) “I felt as though


I were actually in the remote environment as I performed the task”; and (b) “The expe-rience involved unity or fusion of self with the remote environment.” A 7-point ratingscale was associated with each item and was used to capture the degree to which a useragreed with the statements (i.e., subjective ratings of telepresence were made).

The NASA-Task Load indeX (TLX) (Hart & Staveland, 1988) was used to measurethe subjective workload. Subjects completed a subjective comparison of demand factors(mental, physical, temporal, performance, frustration, and effort) once before the begin-ning of the experiment test trials and rated perceived workload at the end of each test trialalong each of the six dimensions. The rankings and ratings of the various demand com-ponents were used to compute a composite index of workload for the telerover navigationtask (a weighted sum of the ratings across all demands).

2.2.4. Subjects. Thirty-two subjects were recruited from the graduate and undergrad-uate student population at North Carolina State University for participation in this studyon a voluntary basis. There were 29 male subjects and 3 female subjects. The average agewas 23.72 years. All subjects had 20/20 vision without correction.

2.2.5. Experiment. The LOA was used as a subject-grouping variable. Each subjectwas assigned to only one control mode with an equal number of subjects experiencingteleoperation control or the telerobotic control mode. The LOA was manipulated as abetween-subjects variable to limit the potential of training carry-over effects from onemode of control to another.

The types of delay and adaptation conditions were manipulated as within-subjects vari-able. Each subject was exposed to all combinations of the three different delays (no delay[control condition], constant and random delays) and the two adaptation settings. Theentire experimental design was replicated once. Thus, two trials were conducted undereach delay–adaptation combination producing 10 trials per subject. One hundred and sixtytrials were completed under both the teleoperation and telerobotic control modes.

2.2.6. Procedures. Experimental procedures were developed for both training and test-ing sessions. During the training session, subjects were introduced to the study and labequipment. This was followed by their informed consent and an anthropometric data sur-vey. The subjects were then familiarized with the different displays as part of the VEinterface and a practice trial was provided with no obstacles appearing in the VE. Fol-lowing the practice trial, subjects completed a Simulator Sickness Questionnaire (SSQ;Kennedy, Lane, Berbaum, & Lilienthal, 1993), which was used as a baseline reading.Subsequently, they were allowed a practice trial under the no delay condition, which wasalso followed by another SSQ. Finally, the subjects were familiarized with the telepres-ence (PQ) and NASA-TLX questionnaires and received a 5-minute break.

During the test session, each subject experienced 10 trials of approximately 5–7 min-utes with 2-minute breaks in between and a 5-minute break midway through the trials.The NASA-TLX and telepresence questionnaire were administered after every trial. Atthe end of 5th and 10th trial, the SSQ was administered.

The entire training procedure took 45– 60 minutes, while the testing procedure took90–120 minutes. Consequently, subjects were recruited to participate for a maximum of3 hours to complete the experiment.

2.2.7. Hypotheses. It was expected that when there was an increase in the controldelay/lag, performance would degrade, or TTC and the number of errors would increase,


with a corresponding decrease in presence and increase in workload. To offset the impactof lag on user performance and perceived presence, the concept of gain adaptation wasexplored. When network delays increased, the telerover controller would automaticallyadapt its gain (or speed) to maintain safe and accurate performance and system stability.This adaptation of gain was expected to increase TTC, but limit the number of perfor-mance errors (collisions) in comparison to conditions involving no adaptation. It was alsohypothesized that when there was deterioration in the network QoS, adapting the gainwould result in a less significant decrease in presence ratings than when no adaptationwas used to account for the lag. Thus, the adaptation conditions were expected to result inhigher presence ratings compared to the no adaptation conditions.

It was also expected that changes in telepresence might vary between the teleoperationand telerobotic control modes. In the telerobotic mode, since the user specified a targetdestination and supervised the telerover actions, the impact of network delay on the userwas expected to be minimal. The user did not directly control the motion of the teleroverand, thus, would only see a slight decrease in navigation speed. In the teleoperation mode,the user directly controlled the telerover and, hence, when there was an adaptation tonetwork QoS deterioration, the user perceived a drop in navigation speed. This wouldincrease task completion time, and was expected to cause some user frustration with sys-tem performance resulting in reduced presence and increased workload. Riley and Kaber(2001) previously observed a negative correlation between user subjective perception oftask frustration (as a demand factor in workload) and telepresence ratings in a teleoper-ation simulation. It was hypothesized that subjects using telerobotic control would expe-rience less deterioration of presence and lower workload compared to those usingteleoperation control.

It was expected that the study would provide insight into the effectiveness of the adap-tation scheme and VR interface for facilitating performance and presence. The experi-ment was also expected to provide insight into the relationships between telepresence,performance, and workload under the various teleoperation test conditions.

3. RESULTS

A multiway analysis of variance (ANOVA) was applied to the dependent variables toinvestigate the influence of LOA, delay type, and gain adaptation on task perfor-mance (TTC and errors), telepresence, and workload. Two statistical models were usedfor the analyses: A full model served as a basis for comparisons between the tele-operated and telerobotic control modes under the specific lag and adaptation conditions(e.g., random with no adaptation). Since adaptation was not relevant to the no-delayconditions, it was removed from the dataset investigated with this model. A reducedstatistical model was also used in which the no-delay control condition was comparedwith the random and constant delay conditions, with and without adaptation, by com-bining the delay type and adaptation into a single independent variable called networkcondition (NC).

Correlation analyses were also conducted using Pearson product-moment correlationcoefficients to identify any significant relationships among TTC, number of errors, tele-presence, and workload.

3.1. Performance

3.1.1. Time-To-Task Completion. The results of an ANOVA on the full statisticalmodel revealed significant main effects of LOA, delay, and adaptation on TTC (see Table 1


for F-test and p values). Average TTC was higher for the teleoperation mode compared tothe telerobotic control mode. The TTC for the constant delay mode was greater than forthe random delay mode and it was higher for the adaptation versus no adaptation mode.The lower TTC under the random delay condition, compared to the constant delay mode,was attributed to the random delay generator, which produced delays between 750 msand 1250 ms. It was observed that the majority of the delays generated under the randomcondition were lower than 1000 ms (i.e., lower than the constant delay setting) and, hence,a lower TTC occurred under the random condition.

Similar ANOVA results were obtained with the reduced statistical model, which includedthe no delay (ND) control condition. The LOA and NC were found to significantly influ-ence TTC (see also Table 1 for F-test and p values). There were also significant individ-ual differences among subjects within the automation groups. In addition, a significanttwo-way interaction of LOA and NC (F (4,319) � 2.86, p � 0.05) was present. Figure 3shows the TTC across the two control mode conditions under the various network set-tings including the ND condition.

Tukey’s HSD procedure was used to further analyze the significant interaction andrevealed the random delay with adaptation (RA) and constant delay with adaptation (CA)conditions to produce the worst TTC among all conditions except the constant delay with

TABLE 1. Significant Main Effects of TTC in Full and Reduced Statistical Models

TTC Full model Reduced model

LOA F (1,255) � 44.61 p � 0.0001 F (1,319) � 45.13 p � 0.0001Delay F (1,255) � 8.54 p � 0.01 n/a n/aAdaptation F (1,255) � 8.88 p � 0.01 n/a n/aNetwork condition n/a n/a F (4,319) � 20.68 p � 0.0001

Figure 3 Average time-to-task completion under various network conditions grouped by level ofautomation.


no-adaptation (CNA) condition under the teleoperation mode. On average, the ND con-dition under the telerobotic mode produced the best TTC, but it was not significantlydifferent from the ND in teleoperation mode and the random delay with no-adaptation(RNA) and random with adaptation (RA) conditions under the telerobotic mode. It wasobserved that regardless of the delay type or adaptation condition, the telerobotic moderesulted in higher performance in terms of TTC as compared to the teleoperation mode.

3.1.2. Number of Collisions/Errors. The ANOVA results on the full statistical modelrevealed LOA, the delay, and adaptation manipulations to significantly influence the num-ber of task-related errors (see Table 2). Similar results were obtained with the reducedmodel, considering the ND control condition (see Table 2 and Figure 4). Significantlyfewer collisions/errors occurred under the teleoperation mode than the telerobotic mode.This might have been because the subjects had complete control over the speed of therover under the teleoperation mode, while in the telerobotic mode they shared controlwith the computer system. In addition, the teleoperation mode provided the subjects withthe capability to stop the rover completely, but this was not available in the teleroboticmode since the rover only stopped if it reached the target destination as defined by thevirtual crosshairs in the aerial view, or if it hit an obstacle. It is possible that the verynature of the automation of rover control could have contributed to the increased errorsunder the telerobotic mode. As Parasuraman, Sheridan, and Wickens (2000) have noted,automation often does not simply supplant the human operator, but it is implemented toaid human performance and, in so doing, may fundamentally change the manner in whichoperators behave. The constant delay condition resulted in a higher number of errors thanthe random delay. Adaptation reduced the number of errors by almost 50% compared tono adaptation condition. The magnitude of error reduction due to adaptation was morepronounced under the constant delay than the random delay condition. No significantinteraction was observed between LOA and NC, as in the analysis of TTC.

3.1.3. Advantages of Gain Adaptation. Figure 5 shows the percentage increase inTTC because of using adaptation over the no-adaptation condition and the correspondingreduction in the number of errors. The plot shows the average increase in TTC underboth of the LOAs and corresponding substantial decreases in the number of errors forboth constant and random delay conditions when using adaptation. Upon initial inspec-tion of the plot, we observed that the effect of gain adaptation is more pronounced underthe constant delay mode compared to the random delay mode. This means the ratio of thepercentage decrease in errors to the percentage increase in TTC is higher for the constantdelay as compared to the random delay under both LOAs (the higher the ratio, the greaterthe performance improvement). This can be attributed to the nature of the random delay,

TABLE 2. Significant Main Effects of Errors in Full and Reduced Statistical Models

Errors Full model Reduced model

LOA F (1,255) � 47.08 p � 0.0001 F (1,319) � 51.34 p � 0.0001Delay F (1,255) � 4.91 p � 0.05 n/a n/aAdaptation F (1,255) � 29.19 p � 0.0001 n/a n/aNetwork Condition n/a n/a F (4,319) � 4.91 p � 0.0001


where the delay varied between 750 ms and 1250 ms as compared to the constantdelay, which was constant at 1000 ms. It can also be noticed that the increase in TTC forthe telerobotic mode was very negligible, as compared to the teleoperation control mode.The resulting reduction in the number of errors across LOA was comparable.

Figure 4 Average collisions for different network conditions grouped by level of automation.

Figure 5 Effect of gain adaptation on time-to-task completion and errors under the two levels ofautomation.


3.2. Telepresence

An ANOVA was conducted on the ratings for each item as part of Draper and Blair’s(1996) measures. It was observed with the reduced statistical model that responses to thefirst statement (PQ1) were significantly influenced by LOA (F (1,319) � 5.88, p � 0.05)with no apparent affect of network condition. On average, PQ1 (telepresence) ratingswere 4.4% greater for the telerobotic control mode, as compared to the teleoperation mode,under all network conditions. The mean telepresence ratings were slightly greater for theadaptation conditions than the no adaptation conditions under the telerobotic mode; how-ever, there was no significant interaction of the LOA and NC manipulations.

3.3. Workload

An ANOVA on the reduced statistical model, allowing for evaluation of the no delaycontrol condition data, indicated that LOA was significant (F (1,319) � 4.34, p � 0.05) ineffect on the overall workload score. There was no apparent affect of the network con-dition. In addition, there were significant individual differences in the workload response.The average workload index was marginally greater (3.7%) for the teleoperation controlmode compared to the telerobotic control mode across the NCs. There was no observabletrend on the adaptation and no adaptation conditions, which was expected based on thelack of a significant NC main effect.

4. DISCUSSION

The results of this study indicate that teleoperation control with gain adaptation for lag ismore effective for maintaining system safety, in part, because of operator direct manualcontrol. The telerobotic control mode tested in this study did not provide for human con-trol intervention between the time a rover destination was targeted and rover navigationto the destination. This increased the number of collisions that could be avoided by usercontrol. The telerobotic mode was faster for completing the task than the teleoperationmode because of less human interaction with the rover and a reduced impact of individualbehavior (i.e., conservative or risky control of speed and direction). Telepresence washigher under the telerobotic mode and this may have been due to the sharing of workloadbetween the human and machine. Riley and Kaber (2001) said that automating compo-nents of a teleoperation task might free-up user mental resources, and increase telepres-ence and performance. The telerobotic control mode in this study freed-up operatorattentional resources, which could be used to perceive the VR interface and might haveresulted in a higher sense of telepresence. This inference is further supported by the lowerobserved workload ratings for telerobotic control compared to teleoperated control.

The constant delay condition produced worse performance than the random delay. Thismay have been because the actual mean random delay generated by the VR system was960 ms with a standard deviation of 140 ms compared to the constant delay of 1000 ms.The gain adaptation algorithm was useful in promoting better performance under bothconstant and random delay conditions compared to the no-adaptation condition. Eventhough not significant, there was a trend of increased telepresence ratings under adapta-tion compared to no-adaptation condition, when using the telerobotic control mode.

Results of the correlation analyses, revealed a significant negative linear associationbetween TTC and telepresence (r � �0.13663, p � 0.05) (i.e., a positive relationship ofthe construct of performance with telepresence). The TTC was found to decrease with an


increase in telepresence ratings. This finding suggests that telepresence has a positiveimpact on human performance in telerover navigation with a VR interface. Riley andKaber (1999) also found a significant positive correlation of subjective telepresence andperformance in their study of simulated telerover control. In the current study, telepresencewas also significantly positively correlated with workload (r � 0.20236, p � 0.001) imply-ing, in general, that increases in workload may result in higher user perception of telep-resence. However, the lower workload control condition (telerobotic control) yielded highertelepresence. Further investigation of the telepresence ratings and perceived workloadscores within control mode (teleoperation or telerobotic) revealed a significant positivecorrelation of the TLX scores with subjective telepresence only under teleoperation con-trol. These results suggest that the teleoperation control condition may have been moresensitive to workload changes due to the delay condition manipulations. The subjectiveratings of workload may have also been sensitive to this and corresponded with telepres-ence ratings. It is possible that as delay conditions became more extreme under teleop-erated control, subjects became more engaged in the task and rated their perceivedinvolvement in the VE as being higher.

A positive relationship between workload and telepresence was also observed by Draperand Blair (1996) during their study of a pipe-cutting operation using a teleoperator. Maand Kaber (2003) observed a similar relationship in an experiment investigating the tele-presence and workload effects on VE design feature manipulations (viewpoint, immer-sion, realism, and audio cues) in a virtual basketball simulation. However, Riley and Kaber(2001) found telepresence to be negatively correlated with subjective workload in a sim-ulation of telerobot-assisted landmine disposal. As argued by Draper and Blair (1996),telepresence may result in the expenditure of attentional resources resulting in increasedcognitive workload. This argument is supported by positive correlations of telepresenceand workload ratings. However, Ma and Kaber (2003) said that there might be some min-imum level of workload required to cause telepresence and that workload above a certain,high level could result in deterioration of telepresence. This may have been the case forthe high task difficulty condition studied by Riley and Kaber (2001) but not their mediumand low difficulty levels (i.e., medium and high mine density), which were not signifi-cantly different. With respect to the results of the present study, the telerobotic mode mayhave imposed a workload engaging users in the task but leaving some disposable atten-tional resources, ultimately resulting in higher telepresence ratings and lower perceptionsof workload. While in the teleoperation mode, higher workload was observed which mighthave been high enough to cause deterioration in average telepresence ratings compared totelerobotic control.

5. CONCLUSION

This study provided evidence that teleoperation control may be more useful for error-sensitive situations, but not time-sensitive situations, in an Internet-based teleoperationscenario. While, telerobotic control is appropriate for tasks demanding lower task com-pletion time, higher workload or higher user presence. The study established that gainadaptation has the potential to improve user performance and system safety and promoteenhanced telepresence experiences under high “end-to-end” teleoperation lag conditions.The results may be applicable to error-sensitive operations like telesurgery, remote han-dling of nuclear waste, and time-sensitive operations like telerobot-assisted search andrescue, sentry, and surveillance.


ACKNOWLEDGMENTS

This research was supported by a grant from the National Science Foundation (NSF)entitled, “CAREER: Telepresence in Teleoperations” (Grant No. IIS-0196342). Theopinions expressed in this paper are those of the authors and do not necessarily reflect theviews of the NSF.

APPENDIX

Definitions of Expert Acronyms

ANOVA: Analysis of varianceCA: Constant delay with adaptationCAN: Constant delay no-adaptationLOA: Level of automationLQG: Linear quadratic GaussianNC: Network conditionND: No delayPQ: Presence QuestionnaireQoS: Quality of serviceRA: Random delay with adaptationRNA: Random delay with no-adaptationSSQ: Simulator sickness questionnaireTCP: Transmission control protocolTLX: Task load indexTTC: Time-to-task completionUDP: User datagram protocolVE: Virtual environmentVR: Virtual reality

REFERENCES

Bejczy, A.K., Kim, W.S., & Venema, S.C. (1990). The phantom robot: Predictive displays for tele-operation with time delays. In the Proceedings of the IEEE International Conference on Robot-ics and Automation—ICRA ’90 (pp. 564–551). Piscataway, NJ: IEEE.

Draper, J.V., & Blair, L.M. (1996). Workload, flow and telepresence during teleoperation, In theProceedings of the IEEE International Conference on Robotics and Automation (Vol. 2, pp. 1030–1035). Piscataway, NJ: IEEE.

Draper, J.V., Kaber, D.B., & Usher, J.M. (1998). Telepresence. Human Factors, 40(3), 354–375.Eberst, C., Stoffler, N.O., Barth, M., & Farber, G. (1999). Compensation of time delays in telep-

resence applications by photo-realistic scene prediction of partially unknown environments. Inthe Proceedings of International conference on Robotics and Applications—IASTED ’99 (pp. 163–168). Calgary, Canada: ACTA Press.

Fei, A., Pei, G., Liu, R., & Zhang, L. (1998). Measurement of delay and hop-count on the Internet.In the Proceedings of IEEE Globecom ‘98—Internet Mini-Conference. Piscataway, NJ: IEEE.

Hart, S.G., & Staveland, L.E. (1988). Development of a multi-dimensional workload rating scale:Results of empirical and theoretical research. In P.A. Hancock & N. Meshkati (Eds.), Humanmental workload (pp. 139–183). Amsterdam: Elsevier.


Hine, B., Hontalas, P., Fong, T., Piguet, L., Nygren, E., & Kline, A. (1995). VEVI: A virtual envi-ronment teleoperations interface for planetary exploration. In the Society of Automobile Engi-neers 25th International Conference on Environmental Systems. Warrendale, PA: Society ofAutomobile Engineers.

Hu, H., Yu, L., Tsui, P.W., & Zhou, Q. (2001). Internet-based robotic systems for teleoperation.International Journal of Assembly Automation, 21(2), 143–151.

Kaber, D.B., Riley, J., Zhou, R., & Draper, J.V. (2000). Effects of visual interface design, controlinterface type, and control latency on performance, telepresence, and workload in a teleopera-tion task. In the Proceedings of the XIVth Triennial Congress of the International ErgonomicsAssociation and 44th Annual Meeting of the Human Factors and Ergonomics Society (pp. 503–506). Santa Monica, CA: Human Factors and Ergonomics Society.

Kennedy, R.S., Lane, N.E., Berbaum, K.S., & Lilienthal, M.G. (1993). Simulator sickness ques-tionnaire: An enhanced method for quantifying simulator sickness. International Journal of Avi-ation Psychology, 3(3), 203–220.

Luck, R., & Ray, A. (1990). An observer-based compensator for distributed delays, Automatica,26(5), 903–908.

Ma, R., & Kaber, D.B. (2003). Telepresence, performance and workload in an immersive virtualenvironment and sporting task. In the Proceedings of the XVth Triennial Congress of the Inter-national Ergonomics Association and the 7th Joint Conference of Ergonomics Society of Korean/Japan Ergonomics Society. Santa Monica, CA: Human Factors and Ergonomics Society.

MacKenzie, S., & Ware, C. (1993). Lag as a determinant of human performance in interactivesystems. In the Proceedings of the ACM Conference on Human Factors in Computing Systems—INTERCHI ’93 (pp. 488– 493). New York: ACM.

Nash, E.B., Edwards, G.W., Thompson, J.A., & Barfield, W. (2000). A review of presence andperformance in virtual environments. International Journal of Human-Computer Interaction,12, 1– 41.

Nilsson, J., Bernhardsson, B., & Wittenmark, B. (1990). Stochastic analysis and control of real-timesystems with random time delays. Automatica, 34(1), 57– 64.

Parasuraman, R., Sheridan, T., & Wickens, C.D. (2000). A model for types and levels of humaninteraction with automation. IEEE Transactions on Systems, Man, and Cybernetics Part A: Sys-tems and Humans, 30(3), 286–297.

Riley, J.M., & Kaber, D.B. (1999). Telepresence and performance effects of visual display type andnavigational aid in virtual reality training of telerover navigation. In the Proceedings of ANS 8thTopical Meeting on Robotics and Remote systems (CD-ROM). LaGrange Park, IL: AmericanNuclear Society.

Riley, J.M., & Kaber, D.B. (2001). Utility of situation awareness and attention for describing tele-presence experiences in a virtual telepresence task. In B. Das & W. Karwowski (Eds.), Proceed-ings of the 2001 International Conference on Computer-Aided Ergonomics and Safety (CD-ROM). Rome, Italy: International Ergonomics Association.

Sheridan, T.B. (1992). Musing on teleoperations and virtual presence. Presence, Teleoperators andVirtual Environments, 1(1), 120–126.

Taylor, K., & Trevelyan, J. (1995). A telerobot on the world wide web. In the Proceedings of theNational Conference of Australian Robotic Association (pp. 108–120). Melbourne, Australia:Australian Robot Association.

Tipsuwan, Y. (2003). Gain scheduling for networked control system. Unpublished doctoral disser-tation, North Carolina State University.

Tipsuwan, Y., & Chow, M. (2001). Network-based controller adaptation based on QoS negotiationand deterioration. In the Proceedings of 27th Annual Conference of the IEEE Industrial Elec-tronics Society—IECON ’01. (pp. 1794–1799). Piscataway, NJ: IEEE.

Tipsuwan, Y., & Chow, M. (2003a). Control methodologies in networked control systems. ControlEngineering Practice, 11, 1099–1111.

Tipsuwan, Y., & Chow, M. (2003b) On the gain scheduling for networked PI controller over IPnetwork. In the Proceedings of 2003 IEEE/ASME International Conference on Advanced Intel-ligent Mechatronics (pp. 20–24). Piscataway, NJ: IEEE.

Tipsuwan, Y., & Chow, M. (2004). Gain scheduler middleware: A methodology to enableexisting controllers for networked control and teleoperation. IEEE Transactions on IndustrialElectronics, 51(6), 1218–1227.


Walsh, G.C., Ye, H., & Bushnell, L. (1999). Stability analysis of networked control systems. In theProceedings of the 1999 American Control Conference (pp. 2876–2880). Dayton, OH: Ameri-can Automatic Control Council.

Watson, B., Spaulding, V., Walker, N., & Ribarsky, W. (1997). Evaluation of the effects of frametime variation on VR task performance. In the Proceedings of VRAIS ’97 IEEE Virtual RealityAnnual Symposium (pp. 38– 44). Piscataway, NJ: IEEE.

Yoshizawa, K., Hashimoto, H., Wada, M., & Mori, S.M. (1996). Path tracking control of mobilerobots using a quadratic curve. In the Proceedings of the 1996 IEEE Intelligent Vehicles Sym-posium (pp. 58– 63). Piscataway, NJ: IEEE.


Documents

Control gain adaptation in virtual reality mediated human–telerobot interaction