IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER …Exploring the Benefits of Augmented Reality Documentation for Maintenance and Repair Steven Henderson, Student Member, IEEE, and

Exploring the Benefits of Augmented RealityDocumentation for Maintenance and Repair

Steven Henderson, Student Member, IEEE, and Steven Feiner, Member, IEEE

Abstract—We explore the development of an experimental augmented reality application that provides benefits to professional

mechanics performing maintenance and repair tasks in a field setting. We developed a prototype that supports military mechanics

conducting routine maintenance tasks inside an armored vehicle turret, and evaluated it with a user study. Our prototype uses a

tracked headworn display to augment a mechanic’s natural view with text, labels, arrows, and animated sequences designed to

facilitate task comprehension, localization, and execution. A within-subject controlled user study examined professional military

mechanics using our system to complete 18 common tasks under field conditions. These tasks included installing and removing

fasteners and indicator lights, and connecting cables, all within the cramped interior of an armored personnel carrier turret. An

augmented reality condition was tested against two baseline conditions: the same headworn display providing untracked text and

graphics and a fixed flat panel display representing an improved version of the laptop-based documentation currently employed in

practice. The augmented reality condition allowed mechanics to locate tasks more quickly than when using either baseline, and in

some instances, resulted in less overall head movement. A qualitative survey showed that mechanics found the augmented reality

condition intuitive and satisfying for the tested sequence of tasks.

Index Terms—Industrial, military, user interfaces, virtual and augmented reality.

Ç

1 INTRODUCTION

MAINTENANCE and repair operations represent an inter-esting and opportunity-filled problem domain for the

application of augmented reality (AR). The majority ofactivities in this domain are conducted by trained main-tenance personnel applying established procedures todocumented designs in relatively static and predictableenvironments. These procedures are typically organizedinto sequences of quantifiable tasks targeting a particularitem in a specific location. These characteristics and othersform a well-defined design space, conducive to a variety ofsystems and technologies that could assist a mechanic inperforming maintenance.

Such assistance is desirable, even for the most experi-enced mechanics, for several reasons. First, navigating andperforming maintenance and repair procedures imposessignificant physical requirements on a mechanic. For eachtask within a larger procedure, a mechanic must first movetheir body, neck, and head to locate and orient to the task.The mechanic must then perform additional physicalmovement to carry out the task. Assistance optimizingthese physical movements can save a mechanic time andenergy. Such savings can be significant when performingdozens of potentially unfamiliar tasks distributed across alarge, complex system. Second, navigating and performing

maintenance and repair procedures impose cognitiverequirements. A mechanic must first spatially frame eachtask in a presumed model of the larger environment, andmap its location to the physical world. The mechanic mustthen correctly interpret and comprehend the tasks. Effectiveassistance in these instances can also save the mechanictime while reducing mental workload.

In this article, which includes deeper analysis of ourearlier work [15], we explore how AR can provide variousforms of assistance to mechanics during maintenance andrepair tasks. We begin with a review of related work and acognitive framework for maintenance and repair assistance.Next, we describe the design and user testing of a prototypeAR application (Figs. 1 and 2) for assisting mechanics innavigating realistic and challenging repair sequences insidethe cramped interior of an armored vehicle turret. Ourapplication uses AR to enhance localization in standardmaintenance scenarios with on-screen instructions, atten-tion-directing symbols, overlaid labels, context-setting 2Dand 3D graphics, and animated models. This information iscombined with a mechanic’s natural view of the main-tenance task in a tracked see-through headworn display(HWD) and is primarily designed to help the mechaniclocate and begin various tasks. Our contributions include adomain-specific user study examining professional me-chanics using our system to maintain actual equipment in afield setting. Our user study demonstrates how mechanicsperforming maintenance sequences under an AR conditionwere able to locate tasks more quickly than when using twobaseline conditions. We also document specific instanceswhen the AR condition allowed mechanics to perform taskswith less overall head movement than when using thesebaselines. Finally, we convey the qualitative insights ofthese professional mechanics with regard to the intuitive-ness, ease of use, and acceptability of our approach.

IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 17, NO. 10, OCTOBER 2011 1355

. The authors are with the Department of Computer Science, ComputerGraphics and User Interfaces Lab, Columbia University, 500 W. 120th St.,450 CS Building, New York, NY 10027.E-mail: {henderso, feiner}@cs.columbia.edu.

Manuscript received 9 Feb. 2010; revised 12 May 2010; accepted 12 June2010; published online 29 Oct. 2010.Recommended for acceptance by G. Klinker, T. Hollerer, H. Saito,and O. Bimber.For information on obtaining reprints of this article, please send e-mail to:[email protected], and reference IEEECS Log NumberTVCGSI-2010-02-0039.Digital Object Identifier no. 10.1109/TVCG.2010.245.

1077-2626/11/$26.00 � 2011 IEEE Published by the IEEE Computer Society

2 RELATED WORK

There has been much interest in applying AR to main-

tenance tasks. This interest is reflected in the formation of

several collaborative research consortiums specifically

dedicated to the topic—ARVIKA [12], Services and Training

through Augmented Reality (STAR) [24], and ARTESAS [2].

These and other efforts have resulted in a sizable body of

work, much of which is surveyed by Ong et al. [22].The majority of related work focuses on specific subsets of

the domain, which we categorize here as activities involving

the inspection, testing, servicing, alignment, installation,

removal, assembly, repair, overhaul, or rebuilding of hu-

man-made systems [37]. Within this categorization, assem-

bly tasks have received the most attention. Caudell and

Mizell [6] proposed a seminal AR prototype to assist in

assembling aircraft wire bundles. Subsequent field testing of

this system by Curtis et al. [8] found the prototype performed

as well as baseline techniques, but faced several practical and

acceptance challenges. Reiners et al. [25] demonstrated aprototype AR system that featured a tracked monocularoptical see-through (OST) HWD presenting instructions forassembling a car door. Baird and Barfield [4] showed thatusers presented with screen-fixed instructions on untrackedmonocular OST and opaque HWDs completed a computermotherboard assembly task more quickly than when usingfixed displays or paper manuals.

Tang et al. [34] studied the effectiveness of AR inassembling toy blocks and found users made fewerdependent errors when aided by registered instructionsdisplayed with a tracked stereoscopic OST HWD, comparedto traditional media. An experiment by Robertson et al. [26]discovered that subjects assembled toy blocks more quicklywhile viewing registered instructions on a tracked biocularvideo see-through (VST) HWD than when using nonregis-tered variants. Zauner et al. [40] demonstrated a prototypesystem for employing AR in a furniture assembly task.Qualitative studies by Nilsson and Johansson for a medicalassembly task [20] and by Salonen and Saaski for 3D puzzleassembly [28] suggest strong user support for AR.

Additional work outside the assembly domain includesFeiner et al.’s knowledge-based AR maintenance prototype[11], which used a tracked monocular OST HWD to presentinstructions for servicing a laser printer. Ockermann andPritchett [21] studied pilots performing preflight aircraftinspections while following instructions presented on anuntracked OST HWD. The results of this study demonstratedan undesired overreliance on computer-generated instruc-tions. Smailagic and Siewiorek [33] designed a collaborativewearable computer system that displayed maintenanceinstructions on an untracked opaque HWD. Schwald andLaval [29] proposed a prototype hardware and softwareframework for supporting a wide range of maintenancecategories with AR. Knopfle et al. [17] developed a prototypeAR application and corresponding authoring tool to assistmechanics in removing and installing components, plugs,and fasteners. Platonov et al. [23] developed a similar proof-of-concept system featuring markerless tracking.

1356 IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 17, NO. 10, OCTOBER 2011

Fig. 1. A mechanic in our user study, wearing an AR display andwristworn control panel, performs a maintenance task inside an LAV-25A1 armored personnel carrier.

Fig. 2. (Left) A mechanic wearing a tracked headworn display removes a bolt from a component inside an LAV-25A1 armored personnel carrierturret. (Right) The AR condition in the study: A view through the headworn display captured in a similar domain depicts information provided usingAR to assist the mechanic. (The view through the headworn display for the LAV-25A1 domain was not cleared for publication due to securityrestrictions, necessitating the substitution of images from an alternative domain throughout this paper.)

There is also notable work on the general task of localizinga user’s attention in AR. Feiner et al. [11] used a 3Drubberband line drawn from a screen-fixed label to apossibly offscreen target object or location. Biocca et al.developed the “Attention Funnel” [5], a vector tunnel drawnto a target, similar to “tunnel-in-the-sky” aviation cockpithead-up displays, and showed that it reduced search timecompared to world-fixed labels or audible cues. Tonnis andKlinker [35] demonstrated that an egocentrically alignedscreen-fixed 3D arrow projected in AR was faster at directinga car driver’s attention than an exocentric alternative. Witheret al. [39] compared the performance of various displays tosupport visual search for text in AR (a task supported bylocalization), but did not detect any significant differencesbetween display conditions. Schwerdtfeger and Klinker [30],[31] studied AR attention-directing techniques to help usersfind and pick objects from stockroom storage bins. Theirframe-based technique outperformed static 3D arrows andvariants of the Attention Funnel.

Two aspects of our contributions distinguish them fromthis previous work. First, other than the wire bundleassembly research conducted by Curtis et al. [8], ourresearch is the only project we know of to include aquantitative study of professional users employing AR formaintenance tasks under field conditions. Our work differsfrom the wire bundle assembly research by examining amore diverse set of maintenance tasks (including inspec-tion, alignment, removal, and installation) in a morerestrictive environment using different comparison condi-tions. Second, our work is the first within the maintenancedomain that articulates the potential benefits of AR forreducing head movement.

3 COGNITIVE DESIGN GUIDELINES

Two projects fundamentally influenced the design of ourAR prototype. The first of these, by Neumann and Majoros[19], proposes that manufacturing and maintenance activ-ities fall into two classes: those focusing mainly on thecognitive, or informational, aspects of a maintenance task,and those focusing on the psychomotor, or workpiece,aspects of a task. Example informational phase activitiesinclude directing attention (localization), comprehendinginstructions, and transposing information from instructionsto the physical task environment. Workpiece phase activ-ities include comparing, aligning, adjusting, and otherforms of physical manipulation. Based on our observationsof professional mechanics operating in a variety of settings,we believe AR applications designed for maintenance andrepair tasks should seek to provide specific assistance inboth phases. Moreover, we suspect that the types ofassistance offered in each phase are distinct. Therefore, wehave structured our research (and this article) accordingly.

The second project influencing our prototype proposed aset of design principles for assembly instruction. Heiser et al.[14] derived eight principles for visualizing processes overtime from observing and surveying users assembling a smalltelevision stand. These heuristics extend earlier use of someof these heuristics in automated generation of graphics andAR [10], [11], [32]. As described in Section 4.1, we sought toadhere to this set of principles in designing the user interfacefor our software prototype.

4 INFORMATIONAL PHASE PROTOTYPE

We developed a hardware and software architecture forstudying AR applications for maintenance. This architec-ture allowed us to implement a prototype focusing on theinformational phases of tasks that we evaluated through theuser study described in Section 4.3. We note that ourprototype is a laboratory proof-of-concept system forexploring the potential benefits of AR for supportingmaintenance procedures under field conditions, but is nota production-ready implementation. Therefore, our soft-ware and hardware choices did not have to reflect the needsof a production environment. We have used our prototypeto study United States Marine Corps (USMC) mechanicsoperating inside the turret of an LAV-25A1 armoredpersonnel carrier. The LAV-25 (of which the LAV-25A1 isa variant) is a light wheeled military vehicle, and the turretportion is a revolving two-person enclosed, cockpit-likestation in the middle of the vehicle. The entire turret volumeis approximately 1 cubic meter, but much electrical,pneumatic, hydraulic, and mechanical infrastructure en-croaches from a myriad of directions and in close proximityto the crew’s operating space. A mechanic servicing theturret works while sitting in one of two seats that are eachfixed along the longitudinal axis of the turret. The resultingwork area is approximately 0.34 cubic meters and spans theentire area surrounding the mechanic.

Because we did not have regular access to the vehicle, weused an extensive set of 3D laser scans to create a mostlyvirtual mockup of the turret, which we used in our lab duringdevelopment. We then finalized our design in an actualturret in two separate pilot tests prior to the user study in thereal turret. The first pilot test involved prototype testing withusers at the Marine Corps Logistics Base in Albany, Georgia.This allowed us to refine our design and gather user feedbackabout our interface. The second pilot test involved fourmechanics from the population recruited for the user studydescribed in Section 4.3. These mechanics experienced nearlythe same test procedure as other participants, but their datawere excluded after we modified two tasks to reduce theoverall execution time of our experiment.

4.1 Software

Our AR application software was developed as a gameengine “mod” using the Valve Source Engine SoftwareDevelopment Kit. The engine “player” serves as a virtualproxy for the user and is positioned by location informationfrom the tracking hardware. All virtual content in the ARscene is provided by custom game engine models, GUIelements, and other components. Full resolution stereo videofrom two Point Gray Firefly MV cameras is stretched to thescene back buffer via an external DLL that hooks the gameengine’s instance of the DirectX graphics interface via theWindows Detours library. The entire scene is rendered instereo at 800� 600 resolution with an average frame rate of75 fps. (The effective video frame rate is approximately 25 fps,due to software upscaling of the stereo images from 2� 640�480 to 2� 800� 600.)

At any given point in time, the application assumes astate representing a particular task (e.g., toggling a switch)within a larger maintenance sequence (e.g., remove an

HENDERSON AND FEINER: EXPLORING THE BENEFITS OF AUGMENTED REALITY DOCUMENTATION FOR MAINTENANCE AND REPAIR 1357

assembly). For each task, the application provides fiveforms of augmented content to assist the mechanic:

1. Attention-directing information in the form of 3Dand 2D arrows.

2. Text instructions describing the task and accompa-nying notes and warnings.

3. Registered labels showing the location of the targetcomponent and surrounding context.

4. A close-up view depicting a 3D virtual scenecentered on the target at close range and renderedon a 2D screen-fixed panel.

5. 3D models of tools (e.g., a screwdriver) and turretcomponents (e.g., fasteners or larger components), ifapplicable, registered at their current or projectedlocations in the environment.

Attention-directing graphics follow a general sequence(Fig. 3) that depends on 6DOF user head pose. If the targetcomponent is behind the mechanic, a screen-fixed greenarrow points the user in the shortest rotational direction tothe target. Once the target is within � 90 degree (yaw) of theuser’s line of sight, a tapered red semitransparent 3D arrowappears, directing the user toward the target. The tail of thearrow is smoothly adjusted and placed along the far edge ofthe display at each frame, based on the vector between thetarget and the user’s projected line of sight on the nearclipping plane. This ensures that the arrow provides asufficient cross section for discernment. As the userapproaches the target, the arrow increases in transparencyand eventually disappears and spawns a highlighting effectfor five seconds at the location of the target. Depending ontask preferences and settings, the 3D arrow will reengage ifthe angle between the user’s head azimuth and the directionto target exceeds 30 degree.

For more complex or potentially ambiguous tasks,animated 3D models are added to the user’s view. Theseanimations show the correct movement of tools orcomponents required to accomplish a particular task. Forexample, when mechanics are instructed to remove orinstall fasteners, animated tools demonstrate the correct toolmotion to accomplish the task, as shown in Fig. 2, right.Animation sequences are controlled and synchronized sothey begin when the 3D arrow disappears, and play for afinite period of time (five seconds).

If a mechanic wishes to replay an animated sequence orcontrol its speed, they can use a wireless wristworncontroller, shown in Fig. 1, which serves as the primarymeans for manually interacting with the user interface of

our prototype. The controller uses a custom 2D interfaceapplication written using the Android SDK, and providesforward and back buttons that allow the mechanic tonavigate between maintenance tasks. When viewing taskswith supporting animation, additional buttons and a sliderare provided to start, stop, and control the speed ofanimated sequences. These animation buttons are hiddenfor nonanimated tasks.

As described in Section 3, we leveraged previouslypublished design heuristics [10], [11], [14], [32] in develop-ing the user interface for our software prototype. Based onour experience with our prototype, we highlight severalheuristics discussed by Heiser et al. [14] that we feel areparticularly crucial to the successful application of AR formaintenance and repair. The first of these is the importanceof displaying one diagram for each major step. As shown inearlier work and in our prototype, AR can populate amechanic’s natural view of a task with a large variety ofvirtual content. A potential challenge lies in scoping andorganizing this content to preserve the notion of separableand easy to comprehend “major steps.” In our prototype,we strived to maintain a manageable task structure bydisplaying text instructions and close-up views (as shownFig. 3) similar to what appear in paper manuals, and alsoallowing the mechanic to control the progression of stepswith the wristworn controller.

A second important heuristic highlights the use ofarrows and guidelines to indicate action (e.g., attachment,alignment, and removal). While our interface has the abilityto display such information using registered 3D models(e.g., Fig. 2, right) such models were limited in ourprototype to showing the position of tools or major turretcomponents. As we summarize in Section 5, we feel futureapplications of AR should seek greater use of arrows andguides in the workpiece portion of certain tasks.

Two additional important and related heuristics govern-ing the design of interfaces for maintenance and repair tasksemphasize showing stable orientations in a manner that isphysically realizable, while avoiding changing viewpoints.The use of AR for maintenance and repair can implicitlypromote these heuristics by providing a unified in situ viewof both an assigned task environment and its accompanyinginstructional content. This is demonstrated in our prototypeby the use of tracked 3D models and labels depictingcomponents in starting, transient, and target orientations, asspecified by certain tasks. Likewise, other 3D models showsuggested tool orientations and movements.


Fig. 3. A typical localization sequence in our prototype. (a) A screen-fixed arrow indicates the shortest rotation distance to target. (b) As the userorients on the target, a semitransparent 3D arrow points to the target. (c) As the user reaches the target, the 3D arrow begins a gradual fade to fulltransparency. (d) When the arrow has completely faded, a brief highlighting effect marks the precise target location.

4.2 Hardware

We experimented with two HWDs while developing ourprototype. The display we eventually used for user trials(Fig. 1) is a custom-built stereo VST HWD constructed froma Headplay 800� 600 resolution color stereo gamingdisplay with a 34-degree diagonal field of view (FOV). Wemounted two Point Gray Firefly MV 640� 480 resolutioncameras to the front of the HWD, which were connected to ashared IEEE 1394a bus on the PC. The cameras areequipped with 5 mm microlenses and capture at 30 fps.This application executes on a PC running Windows XPPro, with an NVIDIA Quadro 4500 graphics card. We alsoexperimented with, and initially intended to use, an NVISnVisor ST color stereo OST HWD. We selected this displaybecause of its bright 1;280� 1;024 resolution graphics,60 degree diagonal FOV, and high transmissivity. However,during pilot testing, we discovered that vehicle assemblieslocated directly in front of and behind the seats preventedusers from moving their head freely while wearing therelatively large nVisor HWD. This necessitated use of ourcustom-built HWD in the user study.

Tracking is provided by a NaturalPoint OptiTracktracking system. The turret’s restricted tracking volumeand corresponding occluding structures created a noncon-vex and limited standoff tracking volume, which led us toemploy 10 tracking cameras to achieve ample coverage.Because we were focused on research, rather than practicaldeployment, we were not concerned with the disadvan-tages of adding a large number of cameras to the existingturret. In contrast, future production-ready AR mainte-nance systems might instead use cameras and other sensorsbuilt into the task environment or worn by the maintainer,possibly in conjunction with detailed environment models.

The OptiTrack system typically uses passive retrore-flective markers illuminated by IR sources in each camera.During pilot testing, we discovered that numerous metallicsurfaces inside the turret created spurious reflections.Although we were able to control for all of these withcamera exposure settings or by establishing masked regionsin each camera, these efforts greatly reduced trackingperformance. Therefore, we adopted an active markersetup, using three IR LEDs arranged in an asymmetrictriangle on the HWD. Given the confined space inside theturret, we were concerned that a worker’s head positioncould potentially move closer than the 0.6 meter minimumoperating range of the OptiTrack. However, experimenta-tion revealed that, for any point inside our work area, atleast four cameras could view a user’s head from beyondthis minimum operating range. Moreover, the active markersetup prevented the possibility of IR light from camerasreflecting off the user’s head at close range. The trackingsoftware streams tracking data at 60 Hz to the PC runningthe AR application over a dedicated gigabit ethernetconnection. The tracking application runs on an AlienwareM17 notebook running Windows Vista, with an additionalenhanced USB controller PC Card.

We implemented our wristworn controller using anAndroid G1 phone (Fig. 1). The device displays a simple setof 2D controls and detects user gestures made on a touchscreen. These gestures are streamed to the PC running the

AR application over an 802.11g link. The mechanic attachesthe device to either wrist using a set of Velcro bracelets.

4.3 Informational Phase User Study

We designed a user study to compare the performance andgeneral acceptance of our prototype (the AR condition) tothat of an enhanced version of the system currently used byUSMC mechanics. We also included an untracked version ofour prototype in the study as a control for HWD confounds.Six participants (all male), ages 18-28, were recruited from arecent class of graduates of the USMC Light WheeledMechanic Course in Aberdeen Proving Ground, Maryland.Each graduate had minimal experience with maintenancetasks inside the turret of the LAV-25A1, which is onlyfeatured in a two-hour introductory block of instructionduring the course. Participants categorized their computerexperience as having no experience (one participant),monthly experience (one participant), weekly experience(one participant), daily experience (two participants) orusing computers multiple times per day (one participant).Participants categorized their experience level with mechan-ical systems as either a basic level of experience (twoparticipants), having some experience (two participants), orvery experienced (two participants). We note that theparticipants’ recent status as students studying maintenanceunder instructors with many years of experience might haveled to underreported mechanical experience levels. Twoparticipants identified themselves as requiring contact lensesor glasses, and both determined that the separate left andright eye focus adjustments on the HWD provided adequatecorrection. All participants were right-handed.

4.3.1 Baseline Comparison Technique

In our experiment, we wanted to compare our prototypeagainst current techniques used by USMC mechanics whileperforming maintenance task sequences. These techniquesprincipally involve the use of an Interactive ElectronicTechnical Manual (IETM) [1], a 2D software applicationdeployed on a portable notebook computer carried andreferenced by mechanics while completing tasks. IETMusers browse electronic documents in portable documentformat (PDF) using a specialized reader, an example ofwhich is shown in Fig. 4.

We felt that a comparison against this system would notbe compelling for several reasons. First, the extra timerequired to navigate this software, which affords less user


Fig. 4. Example screen shot from the currently used IETM interface.

control than common PDF readers, is significant. Second,

the perspective views featured in the software are drawn

from arbitrary locations and contain minimal context,

which requires users to browse multiple pages with the

suboptimal interface. As a result, any task completion or

localization metrics would be heavily influenced by the

time required to negotiate the IETM interface.Therefore, we designed and adopted an improved

version of the IETM interface to use as a baseline in the

study. This baseline (the LCD condition) features static 3D

scenes presented on a 19’’ LCD monitor. The monitor was

fixed to the right of the mechanic (who sat in the left seat of

the turret during our experiment), on an azimuth of roughly

90 degree to the mechanic’s forward-facing seated direction.

The LCD was positioned and oriented to reflect how

mechanics naturally arrange IETM notebook computers

while working from the left seat. During each task, the LCD

presents a single static 3D rendered scene. Each static scene,

such as the example shown in Fig. 5, is rendered using the

same engine that generates virtual content for the AR

condition and depicts identical text instructions, 3D labels,

close-up graphics, and animated sequences (if applicable).

Additional 3D models are added to the scene to depict the

central component of interest, as well as important

surrounding context. For each task, static perspective views

were chosen that generally correspond to how each scene

would naturally appear to a user sitting in the left seat. The

FOV for each scene in the LCD condition was widened to

50 degree to approximate the perspectives used in IETMs.

When experiencing the LCD condition during the user

study, mechanics control the displayed scene by manip-

ulating the system state with the wristworn controller.To control for the general effects of wearing a HWD, we

added a third condition featuring an untracked version of our

AR prototype. This HUD (head-up display) condition uses

screen-fixed graphics that depict text instructions and close-

up views identical to those in the AR condition. However, no

localization aids or 3D models were provided with this

condition. While experiencing the HUD condition, partici-

pants wear the same OST HWD worn in the AR condition,

and interact with the application using the same wristworn

controller used in both the AR and LCD conditions.

4.3.2 Tasks

We selected 18 representative maintenance tasks forinclusion in the user study, from among candidates listedin the operator’s manual for the vehicle [38]. Table 1summarizes the selected set of tasks, and Fig. 6 shows theirapproximate arrangement inside the turret. These tasksserve as individual steps (e.g., removing a screw, as shownin Fig. 1) performed as part of a larger maintenancesequence (e.g., replacing a pump). We specifically avoidedadopting an established sequence of tasks to mitigateexperiential influences in the experiment. We selected tasksthat a trained mechanic could perform while sitting in theleft seat, and which could each be reasonably completed in


Fig. 5. LCD condition.

TABLE 1Selected Tasks (with Descriptions Expurgated for Publication)

and Corresponding Pitch and Azimuth Measured from 0.7Meters above the Center of the Left Turret Seat

Fig. 6. Approximate task azimuths and distances as viewed from abovethe turret and looking down. Neighboring task identifiers are separatedby commas.

under five minutes. We also sought to include a diversity oftasks representing various strata within the larger spectrumof maintenance operations [37].

4.3.3 Procedure

A within-subject, repeated measures design was used,consisting of three conditions (AR, LCD, and HUD) and18 maintenance tasks. The experiment lasted approximately75 minutes and was divided into three blocks, with a shortbreak between blocks. Each block consisted of all 18 tasksfor each condition. Block order was counterbalanced acrossparticipants using a Latin square approach to create astrongly balanced design. The task order within blocks wasfixed, with the participants experiencing the same tasks inthe same location across all three conditions. We addressedpotential learning effects by selecting 18 disparate tasks thatwere not part of any existing repair sequence, to preventmemorization of task order. Before each block, theparticipant was shown how to wear the equipment usedin each particular condition. In the AR and HUD conditions,this consisted of fitting and focusing the HWD, with anadditional brief calibration step for the AR condition. In thisstep, the user was instructed to align two crosshairs trackedby the OptiTrack with two corresponding 3D crosshairsregistered at known points in the turret and tracked withfiducial markers. The observer then used keyboard inputsto manually align the transformation between the turret’scoordinate system and that of the HWD (as worn by aparticular user) while observing video of the AR scene.Although this procedure was accurate enough given ourchoice of tasks, it could be improved by employing a morerobust calibration technique, such as the one proposed byRolland et al. [27] and recently implemented by Jones et al.[16]. For the LCD condition, participants donned a light-weight headband affixed with IR LEDs to facilitate thecollection of tracking data. No portion of this apparatusentered the participant’s field of view during the experi-ment. We also note that this tracking approach did notfeature any eye-tracking capability and assumes theparticipant’s gaze is coincident with their head orientation.During the experiment, we did notice some mechanicsglancing at the LCD from oblique angles though thisactivity appeared to be confined to natural eye motionexhibiting during reading. Future studies should attempt toincorporate these finer gaze patterns into comparison ofdisplay conditions.

Before each block, each participant was afforded anopportunity to rehearse the condition using five practicetasks until they felt comfortable. Tools and fastenersrequired for tasks within the block were arrayed on a flatwaist-high structure to the right of the seat and theirlocations highlighted to the participant.

The timed portion of the block consisted of 18 trial tasksdistributed throughout the mechanic’s work area. Each trialbegan when the mechanic pressed the “next” button on thewristworn controller. This started the overall task comple-tion timer, and triggered the presentation of instructionaltext, close-up views, and labels associated with the trialtask. In the AR condition, cueing information (i.e., the red orgreen arrow) was simultaneously activated, prompting theuser to locate the target. The localization time was recorded

when the user positioned their head such that the targetlocation entered and remained within a 200 pixel radius ofthe center of the display for more than one second. In theAR and HUD conditions, a crosshair was displayed to theparticipant to remind them to center each target. In the LCDcondition, which presented static VR scenes for each taskduring the experiment, collected tracking data werereplayed in a discrete event simulation after the experimentto calculate the localization time. Following target localiza-tion, overall task completion timing continued until themechanic gestured on the wristworn controller for the nexttask. The block then proceeded to the next task until theparticipant experienced all 18 tasks.

4.4 Informational Phase Study Results

4.4.1 Data Preparation

We performed several preprocessing steps prior to analyz-ing our results. First, because the tracker coordinate systemwas centered above the left camera of our VST HWD, wetranslated tracking data points to a position coincident withthe center of rotation for the participant’s head. This wasaccomplished by adding a small offset vector v to eachreading, where v was estimated by combining HWDmeasurements with population-specific anthropometricdata from Donelson and Gordon [9] and supplemented byPaquette and colleagues [13].

We then removed spurious points in the recordedtracking and completion time data sets. For tracking data,we applied a moving average filter as defined by Law andKelton [18]. After some experimenting, we selected awindow size of 0.25 seconds, which was applied to all sixdegrees of freedom. For completion time data, we manuallyinspected the task completion time stamps that weretriggered when the subject gestured for the next task usingthe wristworn controller. In several instances, subjects madeaccidental double gestures, then immediately (usuallywithin two seconds) gestured on the “back” button toreload the appropriate task. We identified and removedeight of these instances.

Our final data preparation step involved normalizingposition and orientation data for each subject. Because theHWD was worn differently by each user, the relativeposition and orientation of the tracker to tasks in theexperiment varies by subject. To standardize all subjects to acommon reference frame, we individually normalized eachsubject’s position and orientation data, as suggested byAxholt et al. [3].

4.4.2 Order Effects

We performed an analysis to check for order effects in ourstudy. We applied a 3 (Presentation Order) �18 (Task)repeated measure ANOVA on both task localization andcompletion time and with our participants as the randomvariable. Presentation order failed to exhibit a significantmain effect on localization time (Fð2;34Þ ¼ 0:039; p ¼ 0:962) orcompletion time (Fð2;34Þ ¼ 0:917; p ¼ 0:431). Based on theseresults, and given our small sample size, we note morework is required to draw any conclusions about how ordereffects impacted our experiment design.


4.4.3 Completion Time Analysis

We applied a 3 (Display Condition) �18 (Task) repeatedmeasure ANOVA to task completion time with ourparticipants as the random variable. Using � ¼ 0:05 as ourcriterion for significance, the display condition had asignificant main effect on completion time (Fð2;34Þ ¼ 5:252;p ¼ 0:028). The mean task completion times for eachcondition were 42.0 seconds (AR), 55.2 seconds (HUD),and 34.5 seconds (LCD) and are shown as asterisks in theTukey box-and-whisker [7] plots in Fig. 7. Post hoccomparison with Bonferroni correction (� ¼ 0:05) revealedmean task completion time under the AR condition was76 percent that of the HUD condition, which was notsignificant (p ¼ 0:331). Mean task completion time underthe LCD condition was 82 percent that of the AR condition(not significant, p ¼ 0:51), and 63 percent of that of the HUDcondition (not significant, p ¼ 0:123). The set of mainte-nance tasks used in the study had a significant main effecton completion time (Fð17;34Þ ¼ 8:063; p < 0:001), which weexpected, given the varying levels of effort required toperform each task.

4.4.4 Localization Time Analysis

We applied a 3 (Display Condition) �18 (Task) repeatedmeasure ANOVA on task localization time with ourparticipants as the random variable. Display conditionexhibited a significant main effect on localization time(Fð2;34Þ ¼ 42:444; p < 0:001). The mean task localizationtimes were 4.9 seconds (AR), 11.1 seconds (HUD), and9.2 seconds (LCD), as shown in Fig. 8. Post hoc comparisonwith Bonferroni correction (� ¼ 0:05) revealed that meanlocalization time under the AR condition was 44 percentthat of the HUD condition, which was statistically sig-nificant (p ¼ 0:001) and 53 percent that of the LCDcondition, which was also statistically significant(p ¼ 0:007). LCD mean localization time was 83 percentthat of HUD, which was not statistically significant(p ¼ 0:085). The particular set of selected maintenance tasksused in the study failed to exhibit a significant main effecton localization time (Fð2;34Þ ¼ 1:533; p ¼ 0:103).

4.4.5 Error Analysis

Errors in our experiment were defined as instances when asubject performed a task to completion on the wrong item,and were logged by the observer during the experiment.

Examples of errors included toggling an incorrect switch,removing an incorrect bolt, or inspecting the wrong item. Ingeneral, we found mechanics made few errors, andconfirmed this with a 3 (Display Condition) �18 (Task)repeated measure ANOVA on task errors with ourparticipants as the random variable. Display condition didnot exhibit a significant main effect on total errors(Fð2;34Þ ¼ 1:00; p ¼ 0:410). This corroborates earlier findingsby Robertson et al. [26].

4.4.6 Head Movement Analysis

Our analysis of head movement focused on the range ofhead rotation, rotational exertion and velocity, and transla-tional exertion and velocity. This analysis was confined toonly the localization portion of each task, because it wasdifficult to isolate head movements from overall bodymotion during the hands-on portion of some tasks. In thesetasks, the user remained relatively static during localization,but adopted many different body poses once they began thephysical portion of the task.

A review of descriptive statistics for overall ranges inhead rotation about each axis revealed left and right headrotation about the neck (yaw) was the greatest source ofrotational movement, and generally conforms to the relativetask azimuths shown in Table 1. A comparison of ranges bytask, shown in Fig. 9, provides a more revealing picture ofthe effect of display condition on head yaw. It should benoted that the range information includes transient headmovements between tasks and, thus, intervals are notnecessarily centered on the target task.

Rotational head exertion during each task was estimatedfor each participant by summing the change in head pitch,yaw, and roll Euler angles at each interval of the recordeddata. Rotational velocity during each task was calculated foreach participant by dividing this total rotational exertion ineach axis by the time required to locate the task. Table 2summarizes these statistics. A 3 (Display Condition) �18(Task) repeated measure ANOVA was performed separatelyfor each statistic along each axis, with participants as therandom variable. In this analysis, display condition had asignificant effect on pitch exertion (Fð2;34Þ ¼ 12:206;p ¼ 0:002), roll exertion (Fð2;34Þ ¼ 34:496; p < 0:001), andyaw exertion (Fð2;34Þ ¼ 32:529; p < 0:001). Post hoc compar-isons with Bonferroni correction (� ¼ 0:05) are summarizedin Table 2. For rotational velocity, display condition had a


Fig. 7. Task completion times (seconds) for AR, HUD, and LCD. Anasterisk marks the mean task completion time for each condition.

Fig. 8. Task localization times (seconds) for AR, HUD, and LCD. Anasterisk marks the mean task localization time for each condition.

significant main effect on mean pitch velocity (Fð2;34Þ ¼12:205; p ¼ 0:002), mean roll velocity (Fð2;34Þ ¼ 48:875;p < 0:001), and mean yaw velocity (Fð2;34Þ ¼ 44:191;p < 0:001). Table 2 captures the post hoc comparisons ofmeans with Bonferroni correction (� ¼ 0:05).

Translational head exertion during each task wasestimated for each participant by summing the change ineuclidean distance exhibited between each interval of therecorded data. The result represents the total euclideandistances the head traveled during localization. A 3 (Dis-play Condition) �18 (Task) repeated measure ANOVA testrevealed a significant main effect of display condition ontranslational exertion (Fð2;34Þ ¼ 17:467; p ¼ 0:001). The meantranslational head exertions were 0.25 meters (AR), 0.36 me-ters (HUD), and 0.68 meters (LCD). Post hoc comparisons ofmean translational exertion with Bonferroni correctionrevealed that exertion exhibited with the AR display was69 percent that of HUD (not statistically significant,p ¼ 0:432), and 37 percent that of LCD, which wasstatistically significant (p ¼ 0:022). HUD exertion was 53percent that of LCD, which was statistically significant(p ¼ 0:01). This reduction in head exertion is furtherdepicted in the 2D histogram heat maps of Fig. 10. Theheat maps depict normalized head positions for all usersacross all tasks and reflect a larger overall area required forhead movement in the LCD condition.

Translational head velocity was estimated for eachparticipant by dividing total translational head exertionduring task localization by the time required to locate thetask. A 3 (Display Condition) �18 (Task) repeated measureANOVA test revealed a significant main effect of displaycondition on translational velocity (Fð2;34Þ ¼ 19:907;p < 0:001). The mean translational head velocities were0.05 meters/second (AR), 0.03 meters/second (HUD), and0.08 meters/second (LCD). Post hoc comparisons of means

with Bonferroni correction revealed that the AR displaycondition maintained a translation velocity 1.6 times that ofthe HUD condition, which was not statistically significant(p ¼ 0:057). The LCD translational velocity was 1.7 timesthat of the AR display condition, which was not statisticallysignificant (p ¼ 0:09), and 2.7 times that of the HUDcondition, which was statistically significant (p ¼ 0:007).

4.4.7 Supporting Task Focus

We employed several methods to analyze how well eachcondition supported a mechanic’s ability to remain focusedon a particular task versus looking elsewhere (e.g., referen-cing a manual or IETM). Quantifying a mechanic’s ability tosustain physical and cognitive focus on his or her currenttask is an important research question in the maintenanceand repair domain. Breaking this focus can prolong thelength of the repair. In addition to incurring more time tomove his or her head, the mechanic will also require time toshift their mental model of the task from what they seephysically to what they interpret in any referenceddocumentation. This interpretation process could poten-tially involve several nontrivial steps: visually searchingimages to identify features of interest, matching thesefeatures to points in the real world, mentally transformingobjects from the documentation’s perspective to the realworld, and memorizing supporting information such aswarnings or instructions.

The first method we employed to examine support fortask focus involved estimating the Distance from CenterPoint (DFCP) for each task, as defined by Axholt et al. [3].This measure reflects the average angular distance a trackedbody deviates about a reference point. In our experiment, theDFCP reference point is the vector between the participant’spredominant head pose and each of the 18 evaluated tasks.With this definition, DFCP provides an indicator of the levelof focus maintained by each mechanic during each assignedtask while experiencing each condition.


TABLE 2Rotational and Translation Exertions and Velocities

Fig. 9. Ranges of head rotation (degrees yaw) for all participants acrosseach task. Tasks are stacked in layers. Each task layer shows rangesfor AR (bottom), HUD (middle), and LCD (top).

We calculated DFCP for each subject under all combina-

tions of tasks and display conditions by first defining a

center direction. We estimated this center direction due to

variations in HWD boresight and because participantsviewed tasks from possibly different poses. For the AR and

HUD display conditions, we defined this center direction

as the mean normalized orientation (pitch and yaw)

exhibited by participants during each task. We includeddata from the entire completion interval in this calculation

to provide sufficient sampling for isolation of the task’s

principal viewing direction. In the case of the LCD display

condition, the mean yaw component of head orientation

was not expected to serve as an accurate estimate because

each participant alternated between looking at the task and

looking at the LCD. Therefore, an additional step was

required to identify the principal viewing direction. This

involved examining the distribution of normalized yaw

angles to estimate the primary direction to each task. This

analysis revealed a distinctive bimodal distribution for

tasks compared to corresponding distributions in normal-

ized yaw for the AR and HUD conditions. An example of

the comparison is shown in Fig. 11. We isolated the

direction to each task in the LCD condition by manually

selecting the local optima in each distribution correspond-

ing to each task’s relative location in the turret. This

allowed us to disambiguate the local optima corresponding

to the task from the local optima corresponding to the LCD.After defining a center direction to each task, we next

summed the distance from this central viewing vector to

every pitch/yaw pair in the head tacking. We approximated

these individual distances by calculating the composite

vector formed by the intersection of each yaw and pitch

angle on a unit sphere. Finally, we calculated DFCP by

dividing the sum of each of these approximated distances by

the number of samples. We applied a 3 (Display Condition)

�18 (Task) repeated measure ANOVA to DFCP with our

participants as the random variable. Display condition

exhibited a significant main effect on localization time

(Fð2;34Þ ¼ 1043:6; p < 0:001). The mean DFCP values were

0.183 meters (AR), 0.137 meters (HUD), and 0.703 meters

(LCD). Post hoc comparison with Bonferroni correction

(� ¼ 0:05) revealed that HUD distance from center point was

0.75 times that of AR, which was not statistically significant

(p ¼ 0:16). The AR distance from center point was 0.27 times

that of LCD, which was significant (p < 0:001). The HUD

distance from center point was 0.19 times that of LCD, which

was also significant (p < 0:001).The second method we employed in examining the

amount of time a mechanic spent looking somewhere other

than at the assigned task involved visually inspecting each

subject’s head azimuth trajectory across the entire se-

quence of 18 tasks. We began by first tracing the ideal head

yaw trajectory over the entire sequence. This ideal

trajectory assumes a mechanic will begin the repair

sequence with his or her head oriented in the forward

direction (0 degree azimuth).


Fig. 11. Distribution of normalized yaw angles for AR and LCD for TaskT4. In each plot, the value x ¼ 0 indicates the population’s mean yaworientation.

Fig. 10. 2D Histograms shown as heatmaps of normalized headpositions (viewed from above the turret looking down) across all usersand tasks for AR (top) HUD (center) and LCD (bottom). The scalerepresents the relative number of hits in each bin of a 150� 150 gridcovering the task area. Bin sizes are 0.004 m (X) and 0.003 m (Y).

An ideal repair would then feature a mechanic movinghis or her head systematically to each subsequent taskazimuth (listed in Table 1), stopping at each location tocomplete the workpiece portion of the task. We next createdsimilar plots for each participant that overlaid their yawtrajectories exhibited while completing the task sequenceunder each display condition. To synchronize the plotsacross all participants, we normalized time in each taskinterval for each participant according to the total time spentlocalizing and completing each task. The resultant plot,shown in Fig. 12, offers some interesting insights aboutpotential interruptions in a mechanic’s task focus. Note, weelected to show only the AR and LCD yaw trajectories here,which were the most interesting, in order to promotereadability (the characteristics of the omitted HUD trajec-tories roughly reflected those of the AR trajectory for eachparticipant). An examination of the plot reflects a distinctiveaperiodic pulse in the LCD yaw trajectory for eachparticipant. This pulse, as confirmed by a careful review ofvideo recorded during the experiment, reflects the momentsduring each task where the mechanic glanced at the LCD.We note it is difficult to statistically quantify this motion dueto possible variations in the position of each mechanic’s headthroughout the task. However, we believe the distinctivesignal of the LCD trajectory roughly approximates thenumber of times the mechanic turned his head to glance atthe LCD. Visually comparing the LCD yaw trajectories tothose of AR appears to indicate the AR condition allowedmechanics to remain more focused on the task at hand.

4.4.8 Qualitative Results

We asked each participant to complete a postexperimentquestionnaire. This questionnaire featured five-point Likertscale questions (1 ¼ most negative; 5 ¼ most positive) toevaluate ease of use, satisfaction level, and intuitivenessfor each display condition. The summary results from theseratings, shown in Fig. 13, are difficult to generalize, givenour small population size and individual rating systems. In

terms of ease of use, median response for LCD (5) washighest, followed by AR (4.5) and HUD (3.5). For ratingsatisfaction, median response to AR (5) was highest,followed by LCD (4) and HUD (4). For rating intuitiveness,median response to AR (4.5) tied with LCD (4.5), followedby HUD (4).

Fig. 14 (top) depicts the distribution of responses whenwe asked participants to rank the techniques in order ofpreferred use. The figure shows four of the six participantsranked the LCD condition first. A Friedman test indicatedthis was a significant ranking (�2ð6; 2Þ ¼ 7:0; p ¼ 0:03).Subsequent pair-wise Wilcoxon tests revealed that LCDwas ranked significantly better than HUD (p ¼ 0:02). Fig. 14(bottom) shows the distribution of responses when weasked participants to rank the techniques as to how intuitivethey were. This distribution shows four of the sixparticipants ranked the AR condition first. However, aFriedman test indicated this was not a significant ranking(�2ð6; 2Þ ¼ 4:33; p ¼ 0:12).

We also asked participants to comment on each displaycondition. In reviewing the LCD condition, participantswere nearly unanimous in their appreciation for the system.For instance, P1 reported “I liked being able to see what I wasdoing on the screen. I think the screen idea is good, because itdoesn’t restrict your light or head movements.” P2 added “It wasa lot easier to look at a screen than to have your vision blocked bythe popups on the screen,” which offers insights into perceivedocclusion issues resulting from virtual content in the ARand HUD conditions. Interestingly, none of the participants


Fig. 12. Head orientation (yaw) trajectories for each use under AR andLCD conditions. The X-axis shows normalized elapsed time for eachtask, and the Y-axis shows rotational head displacement about theforward facing direction.

Fig. 13. Survey response histograms by condition for ease of use (top),satisfaction (middle), and intuitiveness (bottom). Median values for eachcondition are shown as triangles.

commented on the disadvantage of having to look back andforth from the target task to the LCD screen. Conversely,several participants actually highlighted the LCD condi-tion’s ability to help them localize. P4 offered “I liked theLCD the most with the program showing me right where the partwas, and what tool, without the headgear getting in the way.” P3confirmed “things were easy to find” with the LCD.

When commenting on the AR condition, the participantsoffered useful feedback on our choices of visual assistance. Indescribing the 3D attention-directing arrow, P1 wrote “Ienjoyed this system the most . . .was easy to navigate with thetracking red line.” P1 also commented on our use of overlaidvirtual models, adding “The 3D image indicators were mostsatisfying, which allowed for proper item location.” P6 also foundthe attention-directing graphics of our interface helpful,writing “Prior systems may use over-technical terms that cansometimes be confusing, however this system is directive and simplypoints. I find that feature extremely helpful.” While echoing thesesame sentiments about attention-directing graphics, P5offered additional feedback about the use of animation,“The lines pointing to the objects make it very easy...the animationof the wrench going a different direction, whether tightening orloosening is nice.” P2 found the close-up view helpful inmitigating registration issues when stating “the ‘red line’ takesyou right to what your are looking on . . . the only problem I had wasthe arrow sometimes didn’t point to exactly what I was working onbut the close-up view helped sort out any confusion.”

Several participants commented on negative aspects ofthe AR condition. P3 offered “the red line blocked my line ofsight.” P4 described the red 3D arrow as “in the way,” butyielded that the arrow “would help someone who has very littleor no experience.” Despite our efforts to control for occlusionby fading the 3D arrow once the mechanic oriented on thetarget task, these later two comments suggest more carefulwork is needed to prevent occlusion during localization.

Participant reaction to the HUD condition was over-whelming negative. P1 wrote “My least favorite systembecause I didn’t know where some things were located in theturret. . . Identification was more difficult for someone not beingcompletely familiar with the turret.” P3 described thisexperience with HUD as “It wasn’t hard but it was a littleconfusing and when it got confusing it got frustrating. It took meawhile to find what I was looking for.” Despite the fact that theHUD condition afforded the same visual acuity andfreedom of movement as the AR condition, severalparticipants singled out these characteristics only whileexperiencing the HUD condition. P2 offered “it restrictedsome of my head movements in the vehicle and the screen was alittle dark and made things kind of hard to see.” P4 cited “pictureand head clearance” as drawbacks with the HUD condition.

When we asked the participants to list additionaltechnologies that might assist with their roles as mechanics,we received several interesting ideas:

1. P1: “Perhaps a device that lets you be able to see usingyour peripheral vision. . . maybe just use one eye.”

2. P2: “I think a voice activated system would make thingseasier because it would be completely hands free.”

3. P4: “Better picture quality and smaller head gear.”4. P5: “Virtual wiring diagram and a hydraulic diagram.”5. P6: “Perhaps an audio track that gives the instructions

along with the visual aids. I also think if the software couldinterpret the movements and actions it could acknowledgewhen the task was completed and give advice.”

5 LESSONS LEARNED

Our experience with the user study led us to derive thefollowing lessons learned in applying AR to the main-tenance and repair domain:

AR can reduce the time required to locate a task. Astatistically significant result showed the AR displaycondition allowed mechanics to locate tasks more quicklythan when using the LCD condition (representing animproved version of the IETMs currently employed inpractice).

AR can reduce head and neck movements during a

repair. An additional statistically significant result of thestudy showed that the AR display condition allowedmechanics to experience less head translation and rotationthan the LCD display condition during task localization.Descriptive statistics show that, in general, subjects experi-encing the AR condition also required smaller ranges ofhead movement. These results highlight an ability topotentially reduce overall musculoskeletal workloads andstrain related to head movement during maintenance tasks.However, more work is required to reconcile strainreductions resulting from less movement with the addedstrain of wearing a HWD. A technique proposed by Tumleret al. [36], which uses heart rate variability to measurestrain, could be useful for this analysis.

More emphasis is needed on workpiece activities. Theuser study revealed a lack of statistical separation betweenthe overall mean tasks completion times for the AR andLCD display conditions. We suspect this can be improvedby further developing the AR assistance offered during the


Fig. 14. Survey response histograms by condition for rankings of mostpreferred system (top) and most intuitive (bottom).

workpiece portion of maintenance tasks. As described inSection 3, activities in this phase require visual informationsupporting such actions as adjustment, alignment, anddetection. While we did provide some alignment androuting information (e.g., content showing the positionand movements of select tools and turret components), thiswas common to both AR and LCD. Future work shouldexamine specific visual aids rendered in AR to promotethese workpiece activities.

Mechanics may be willing to tolerate shortcomingswith HWD technology if the HWD provides value.Despite the disadvantage of wearing a bulky, relativelylow-resolution prototype VST HWD with fixed focuscameras, and a narrow FOV, participants rated the ARcondition at least as favorably as LCD in terms ofsatisfaction and intuitiveness. While some participantsacknowledged the visibility constraints experienced whileusing AR, they tempered this critique with appreciation forthe assistance it offered.

Mechanics are sensitive to occlusions caused by aug-mented content. As described in the user comments inSection 4.4.8, several of the same mechanics who praised theeffectiveness of AR visuals were also bothered when thesame content occluded a task being performed. We suspectthat this trade-off between the positive and negative aspectsof overlaid AR content is likely related to the experience-level of the individual mechanic. Future AR applications inthis domain should tailor AR content to address the needsof each individual mechanic. Likewise, applications shouldadopt a “do no harm” rule of thumb, and limit anyassistance to only what is required. Where possible, ARapplications should include interaction techniques to allowmechanics to easily dismiss content once it is no longerneeded (e.g., labels) and control the speed of “fade-away”animations (e.g., our red 3D attention-directing arrow).

6 CONCLUSIONS

We described a research prototype AR application formaintaining and repairing a portion of a military vehicle,and presented the results of a controlled user study withprofessional mechanics in a field setting. The AR applica-tion allowed mechanics to locate individual tasks in amaintenance sequence more quickly than when using animproved version of currently employed methods. Theprototype AR application also resulted in mechanicsmaking fewer overall head movements during task locali-zation and was favorably received by study participants interms of intuitiveness and satisfaction.

We were especially encouraged to achieve these resultswith a population of professionally trained mechanicsworking in a field setting, who expressed support for ourapproach. In the future, we plan to conduct a similarevaluation with an improved prototype in an effort todetermine whether additional benefits can be provided byAR in the workpiece portion of maintenance and repair tasks.

ACKNOWLEDGMENTS

This research was funded in part by ONR Grant N00014-04-1-0005, US National Science Foundation (NSF) Grant

IIS-0905569, and generous gifts from NVIDIA and Google.The authors thank Bengt-Olaf Schneider, who provided theStereoBLT SDK to support the display of stereo cameraimagery, and Kyle Johnsen, who advised on use of theOptiTrack system. The authors are grateful for theassistance of cadre and students at Aberdeen ProvingGround, as well as the support of engineers at the MarineCorps Logistics Base including Mike Shellem, CurtisWilliams, Andrew Mitchell, and Alan Butterworth. Theyalso thank David Madigan and Magnus Axholt for insightsshared during design and analysis of our experiment.Permission to use the LAV-25A1 photos shown in thispaper was granted by the USMC LAV Program Manager.

REFERENCES

[1] Interactive Electronic Technical Manuals—General Content, Style,Format and User-Interaction Requirements, Dept. of Defense, 2007.

[2] “ARTESAS—Advanced Augmented Reality Technologies forIndustrial Service Applications,” http://www.artesas.de, May2009.

[3] M. Axholt, S. Peterson, and S.R. Ellis, “User Boresight CalibrationPrecision for Large-Format Head-Up Displays,” Proc. VirtualReality Software and Technology (VRST ’08), pp. 141-148, 2008.

[4] K.M. Baird and W. Barfield, “Evaluating the Effectiveness ofAugmented Reality Displays for a Manual Assembly Task,”Virtual Reality, vol. 4, pp. 250-259, 1999.

[5] F. Biocca, A. Tang, C. Owen, and F. Xiao, “Attention Funnel:Omnidirectional 3D Cursor for Mobile Augmented Reality Plat-forms,” Proc. SIGCHI Conf. Human Factors in Comp. Systems (CHI’06), pp. 1115-1122, 2006.

[6] T.P. Caudell and D.W. Mizell, “Augmented Reality: An Applica-tion of Heads-Up Display Technology to Manual ManufacturingProcesses,” Proc. Hawaii Int’l Conf. System Sciences, vol. 2, pp. 659-669, 1992.

[7] J.M. Chambers, Graphical Methods for Data Analysis. DuxburyResource Center, 1983.

[8] D. Curtis, D. Mizell, P. Gruenbaum, and A. Janin, “Several Devilsin the Details: Making an AR Application Work in the AirplaneFactory,” Proc. Int’l Workshop Augmented Reality (IWAR ’98),pp. 47-60, 1999.

[9] S.M. Donelson and C.C. Gordon, “1995 Matched AnthropometricDatabase of US Marine Corps Personnel: Summary Statistics,”Technical Report TR-96-036, Natick Research Development andEng. Center, 1996.

[10] S. Feiner, “Apex: An Experiment in the Automated Creation ofPictorial Explanations,” IEEE Computer Graphics and Applications,vol. 5, no. 11, pp. 29-37, Nov. 1985.

[11] S. Feiner, B. MacIntyre, and D. Seligmann, “Knowledge-BasedAugmented Reality,” Comm. ACM, vol. 36, pp. 53-62, 1993.

[12] W. Friedrich, “ARVIKA-Augmented Reality for Development,Production and Service,” Proc. Int’l Symp. Mixed and AugmentedReality (ISMAR ’02), pp. 3-4, 2002.

[13] C.C. Gordon, S.P. Paquette, J.D. Brantley, H.W. Case, and D.J.Gaeta, A Supplement to the 1995 Matched AnthropometricDatabase of US Marine Corps Personnel: Summary Statistics,Natick, 1997.

[14] J. Heiser, D. Phan, M. Agrawala, B. Tversky, and P. Hanrahan,“Identification and Validation of Cognitive Design Principles forAutomated Generation of Assembly Instructions,” Proc. WorkingConf. Advanced Visual Interfaces, pp. 311-319, 2004.

[15] S. Henderson and S. Feiner, “Evaluating the Benefits ofAugmented Reality for Task Localization in Maintenance of anArmored Personnel Carrier Turret,” Proc. Int’l Symp. MixedAugmented Reality (ISMAR ’09), pp. 135-144. 2009.

[16] J.A. Jones, I.I.J. Edward Swan, G. Singh, E. Kolstad, and S.R. Ellis,“The Effects of Virtual Reality, Augmented Reality, and MotionParallax on Egocentric Depth Perception,” Proc. Applied Perceptionin Graphics and Visualization, pp. 9-14, 2008.

[17] C. Knopfle, J. Weidenhausen, L. Chauvigne, and I. Stock,“Template Based Authoring for AR Based Service Scenarios,”Proc. IEEE Virtual Reality (VR ’05), pp. 249-252, 2005.

[18] A.M. Law and W.D. Kelton, Simulation Modeling and Analysis.McGraw-Hill Higher Education, 1997.


[19] U. Neumann and A. Majoros, “Cognitive, Performance, andSystems Issues for Augmented Reality Applications in Manufac-turing and Maintenance,” Proc. IEEE Virtual Reality (VR ’98), pp. 4-11, 1998.

[20] S. Nilsson and B. Johansson, “Fun and Usable: Augmented RealityInstructions in a Hospital Setting,” Proc. Australasian Conf.Computer-Human Interaction, pp. 123-130, 2007.

[21] J.J. Ockerman and A.R. Pritchett, “Preliminary Investigation ofWearable Computers for Task Guidance in Aircraft Inspection,”Proc. Int’l Symp. Wearable Computers (ISWC ’98), pp. 33-40, 1998.

[22] S.K. Ong, M.L. Yuan, and A.Y.C. Nee, “Augmented RealityApplications in Manufacturing: A Survey,” Int’l J. ProductionResearch, vol. 46, pp. 2707-2742, 2008.

[23] J. Platonov, H. Heibel, P. Meier, and B. Grollmann, “A MobileMarkerless AR System for Maintenance and Repair,” Proc. Int’lSymp. Mixed and Augmented Reality (ISMAR ’06), pp. 105-108, 2006.

[24] A. Raczynski and P. Gussmann, “Services and Training ThroughAugmented Reality,” Proc. European Conf. Visual Media Production(CVMP ’04), pp. 263-271, 2004.

[25] D. Reiners, D. Stricker, G. Klinker, and S. Muller, “AugmentedReality for Construction Tasks: Doorlock Assembly,” Proc. Int’lWorkshop Augmented Reality (IWAR ’98), pp. 31-46, 1999.

[26] C.M. Robertson, B. MacIntyre, and B.N. Walker, “An Evaluationof Graphical Context When the Graphics are Outside of the TaskArea,” Proc. Int’l Symp. Mixed and Augmented Reality (ISMAR ’08),pp. 73-76, 2008.

[27] J.P. Rolland, C.A. Burbeck, W. Gibson, and D. Ariely, “TowardsQuantifying Depth and Size Perception in 3D Virtual Environ-ments,” Presence, vol. 4, pp. 24-48, 1995.

[28] T. Salonen and J. Saaski, “Dynamic and Visual AssemblyInstruction for Configurable Products Using Augmented RealityTechniques,” Advanced Design and Manufacture to Gain a Competi-tive Edge, pp. 23-32, Springer, 2008.

[29] B. Schwald and B.D. Laval, “An Augmented Reality System forTraining and Assistance to Maintenance in the IndustrialContext,” Proc. Winter School of Computer Graphics, pp. 425-432,2003.

[30] B. Schwerdtfeger and G. Klinker, “Supporting Order Picking withAugmented Reality,” Proc. Int’l Symp. Mixed and Augmented Reality(ISMAR ’08), pp. 91-94, 2008.

[31] B. Schwerdtfeger, R. Reif, W.A. Gunthner, G. Klinker, D.Hamacher, L. Schega, I. Bockelmann, F. Doil, and J. Tumler,“Pick-by-Vision: A First Stress Test,” Proc. Int’l Symp. Mixed andAugmented Reality (ISMAR ’09), pp. 115-124, 2009.

[32] D.D. Seligmann and S. Feiner, “Automated Generation of Intent-Based 3D Illustrations,” Proc. Conf. Computer Graphics andInteractive Techniques, pp. 123-132, 1991.

[33] A. Smailagic and D. Siewiorek, “User-Centered InterdisciplinaryDesign of Wearable Computers,” ACM SIGMOBILE MobileComputing and Comm. Rev., vol. 3, pp. 43-52, 1999.

[34] A. Tang, C. Owen, F. Biocca, and W. Mou, “ComparativeEffectiveness of Augmented Reality in Object Assembly,” Proc.SIGCHI Conf. Human Factors in Comp. Systems (CHI ’03), pp. 73-80,2003.

[35] M. Tonnis and G. Klinker, “Effective Control of a Car Driver’sAttention for Visual and Acoustic Guidance Towards theDirection of Imminent Dangers,” Proc. Int’l Symp. Mixed andAugmented Reality (ISMAR ’06), pp. 13-22, 2006.

[36] J. Tumler, R. Mecke, M. Schenk, A. Huckauf, F. Doil, G. Paul,E. Pfister, I. Bockelmann, and A. Roggentin, “Mobile Aug-mented Reality in Industrial Applications: Approaches forSolution of User-Related Issues,” Proc. Int’l Symp. Mixed andAugmented Reality (ISMAR ’08), pp. 87-90, 2008.

[37] U.S. Army, Maintenance Operations and Procedures (Field Manual 4-30.3), 2007.

[38] U.S. Marine Corps, Light Armored Vehicle Operator’s Manual (LAV-25A1), 2003.

[39] J. Wither, S. DiVerdi, and T. Hollerer, “Evaluating Display Typesfor AR Selection and Annotation,” Proc. Int’l Symp. Mixed andAugmented Reality, pp. 1-4, 2007.

[40] J. Zauner, M. Haller, A. Brandl, and W. Hartman, “Authoring of aMixed Reality Assembly Instructor for Hierarchical Structures,”Proc. Int’l Symp. Mixed and Augmented Reality (ISMAR ’03), pp. 237-246, 2003.

Steven Henderson is a PhD candidate incomputer science in the Computer Graphicsand User Interfaces Lab at Columbia University,where he is researching augmented realityinterfaces for procedural tasks. He is servingas an assistant professor in the United StatesMilitary Academy’s Department of SystemsEngineering, West Point, New York. He is astudent member of the IEEE.

Steven Feiner received the PhD degree incomputer science from Brown University. He isa professor of computer science at ColumbiaUniversity, where he directs the ComputerGraphics and User Interfaces Lab. His researchinterests include augmented reality and virtualenvironments, knowledge-based design of gra-phics and multimedia, mobile and wearablecomputing, games, and information visualiza-tion. He is coauthor of the well-known text

Computer Graphics: Principles and Practice, received an ONR YoungInvestigator Award, and, together with his students, has won best paperawards at UIST, CHI, VRST, and ISMAR. He has been general chair orcochair for the ACM Symposium on Virtual Reality Software andTechnology (VRST 2008), the International Conference on IntelligentTechnologies for Interactive Entertainment (INTETAIN 2008), and theACM Symposium on User Interface Software and Technology (UIST2004); program cochair for the IEEE International Symposium onWearable Computers (ISWC 2003); doctoral symposium chair for ACMUIST 2009 and 2010; and member of the steering committees for theIEEE Computer Society Technical Committee on Wearable InformationSystems, the IEEE International Symposium on Mixed and AugmentedReality, and the ACM Symposium on Virtual Reality Software andTechnology. He is a member of the IEEE.

. For more information on this or any other computing topic,please visit our Digital Library at www.computer.org/publications/dlib.


Documents

IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER …Exploring the Benefits of Augmented Reality Documentation for Maintenance and Repair Steven Henderson, Student Member, IEEE, and