Comparing Projected In-Situ Feedback at the ManualAssembly Workplace with Impaired Workers

Markus Funk1, Andreas Bachler2, Liane Bachler2, Oliver Korn1, Christoph Krieger1,Thomas Heidenreich2, Albrecht Schmidt1

1University of Stuttgart (Pfaffenwaldring 5a, 70569 Stuttgart, Germany)2University of Applied Sciences Esslingen (Kanalstraße 33, 73728 Esslingen am Neckar, Germany)

firstname.lastname@vis.uni-stuttgart.de1 – firstname.lastname@hs-esslingen.de2

ABSTRACTWith projectors and depth cameras getting cheaper, assistivesystems in industrial manufacturing are becoming increasinglyubiquitous. As these systems are able to continuously providefeedback using in-situ projection, they are perfectly suited forsupporting impaired workers in assembling products. How-ever, so far little research has been conducted to understand theeffects of projected instructions on impaired workers. In thispaper, we identify common visualizations used by assistivesystems for impaired workers and introduce a simple contourvisualization. Through a user study with 64 impaired partic-ipants we compare the different visualizations to a controlgroup using no visual feedback in a real world assembly sce-nario, i.e. assembling a clamp. Furthermore, we introduce asimplified version of the NASA-TLX questionnaire designedfor impaired participants. The results reveal that the contourvisualization is significantly better in perceived mental loadand perceived performance of the participants. Further, partic-ipants made fewer errors and were able to assemble the clampfaster using the contour visualization compared to a video vi-sualization, a pictorial visualization and a control group usingno visual feedback.

Author KeywordsAugmented Reality; Impaired Persons; Assistive System;Projection

ACM Classification KeywordsH.5.m. Information Interfaces and Presentation (e.g. HCI):Miscellaneous

INTRODUCTIONTransferring knowledge is one of the most important areasin the manual manufacturing industry. New workers need tobe taught how a product is assembled. Especially when pro-ducing many variants of a product, processes become more

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACMmust be honored. Abstracting with credit is permitted. To copy otherwise, or republish,to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from Permissions@acm.org.PETRA’15, July 01 – 03, 2015, Corfu, Greece.Copyright © 2014 2015 ACM. ISBN 978-1-4503-3452-5/15/07$15.00 .DOI: http://dx.doi.org/10.1145/2769493.2769496

Figure 1. An impaired worker is assembling a clamp with projected in-situ feedback from an assistive system using the contour visualization.

complex and are harder to learn. Usually, new workers aretaught by experienced workers or printed instructions. In somecompanies video tutorials are used to teach workflows. Theseso-called utility movies are e.g. used by Memex-academy1.In the last years projectors became cheaper and affordable,especially by industry standards. Thereby, projecting arbitraryinformation onto physical objects has become possible [17].Recently, the industry adapted to using projectors for display-ing instructions in-situ, i.e. projecting instructions directlyinto the workers field of view. Commercial system are alreadyusing this technology. Light Guide Systems from OPS solu-tions2 uses a projector to display assembly instructions, whileWERKLICHT from Extend3D3 is using a laser to highlightdrilling or welding points.

One of the most important area of application of such assistivesystems is the inclusion of impaired workers4 into the work-ing life [10]. Through continuously providing instructions,

1http://www.memex-academy.eu/ (last access 01-15-2015)2http://www.ops-solutions.com/ (last access 01-15-2015)3http://www.extend3d.de/en/products/wl_pro/ (last ac-cess 01-15-2015)4To counteract stigmatization and inappropriate connotation, the peo-ple with impairment of functional health are referred as ”impairedworkers” in this document. This term refers to a limitation of theperformance, thus it is meant the maximum power level of a per-son regarding a task, or action under test, standard or hypotheticalconditions [20].

1

impaired workers can work on more complex products [19]which enables a better integration. Additionally, projected ele-ments and the hardware setup can be made interactive using adepth-camera [22, 7]. Using this technology, the system canimplicitly react on the workers actions and display appropriatefeedback, or provide interactive controls. Thereby, the sys-tem can provide cognitive assistance for impaired workers inmanual manufacturing without the need of a supervisor. Al-though research focusing on providing gamification elementsin manual manufacturing involving impaired workers has beenconducted [13, 12], the suitability of different visualizations ofprojected instructions for assembly tasks has not been exploredyet.

In this paper, we assess the effects of projected in-situ instruc-tions on the learnability of working processes of impairedworkers. In a study involving 64 impaired workers, we com-pare four different visualizations for displaying projected in-situ instructions. The contribution of this paper is three-fold.(1) We present a simple contour-based in-situ visualization forpresenting working instructions that can easily be understoodby impaired workers (2) We present quantitative and qualita-tive results of a study comparing three different visualizationsfor displaying projected in-situ instructions to a baseline. (3)We introduce a simplified version of the NASA-Task Load In-dex (NASA-TLX) [8] questionnaire that is easy to understandand tailored to be understood by impaired participants.

RELATED WORKSystems for displaying in-situ feedback and providing Aug-mented Reality for impaired persons have been the topic ofvarious research projects. We organize this previous workaccording to their used visualizations and highlight their find-ings.

A general literature review about assistive technology for per-sons with cognitive disabilities is provided by Sauer et al. [19].They conclude that especially for this target group, assistivetechnology can have a positive effect on the person’s perfor-mance and therefore enable building more complex products.This information can be provided by projecting it directly intothe environment where it is needed. This so called in-situ pro-jection was first proposed by Pinhanez [17]. In his everywheredisplay project, a projector and a rotatable mirror were usedto augment physical objects with digital information. In thecontext of searching items, Butz et al. [4] used a rotatablecamera-projector system to first visually identify importantobjects that were equipped with a marker. Later position in-formation can be projected directly onto sought objects byhighlighting them. Zhou et al. [23] use in-situ projection forhighlighting welding spots during welding or afterwards forinspecting the quality. They experiment with different visual-izations of the highlighted spot.

Textual information. In 1992, Cuvo et al. [5] used textualinstructions to teach tasks to persons with mild cognitive dis-abilities. They found, that feedback about the performance isimportant for the workers. More recently, the LuminAR [15]system included a camera and a projector in an anglepoise

lamp. When objects are placed under the lamp, they can beaugmented with additional information. This technique canbe used in a shop window, showing the price and additionalinformation about exhibited articles. In this system, text andimages are used to display the information.

Pictorial information. Compared to text, more widespreadare pictorial instructions as they are language independent anddo not require the user to be able to read. Pictorial instructionsare used for teaching daily life skills to persons with cognitivedisabilities e.g. how to dress themselves [16], how to clean,or how to cook [11]. In a study, Steed et al. [21] investigatedif persons with cognitive disabilities are capable of learningdaily life tasks without being continuously supervised by asocio-educational instructor. After an initial instruction how touse pictorial instructions, the participants had to learn how touse a vacuum cleaner just using the pictorial instructions. Theresults show that using pictorial instructions, the participantswere significantly better. Additionally, participants could re-member the instructions over a long period of time and evenlearn new tasks just using pictorial instructions. Lancioni etal. [14] experimented with pictorial instructions for performingtasks. In a study, they compared instructions on an computer-aided palm device to instructions on cards. Participants usingthe computer-aided palm device to view the pictorial instruc-tions performed better. Also the computer-aided instructionswere preferred by the participants.

Another strand of research focused on how to build easilyunderstandable pictorial instructions [1, 9]. There, researchsuggested building hierarchical pictorial instructions where thereader can see the action that is being performed. Step-by-stepinstructions enable the reader to better identifly the step that isbeing performed. Furthermore, the parts should be orientedin a way that all important features are visible to the reader.A number of pictorial instructions are built complying withthe design guidelines proposed by Agrawala et al [1]. Kornet al. [13] and Bannat et al. [2] also use a camera-projectorsystem that uses a similar concept as our prototype. In theirsystems, both use pictorial instructions in a manufacturingenvironment for assembling LEGO models. The images usedin their project look exactly as the ones in printed manuals.

Video information. Ruther et al. [18] also use a camera-projector system to provide interactive instructions in a sterilearea. In an experiment, they compare interactive projectedinstructions to paper-based instructions. In their prototype, theinstructions are video-based and interactive using a projecteduser interface.

Overall, previous work investigated how textual, pictorial andvideo instructions can be used in the context of assistive sys-tems using in-situ projection. For textual and pictorial instruc-tions, studies with cognitively impaired participants have beenconducted. However, a comprehensive study comparing allproposed visualizations of feedback has not been done yet. Inthis paper, we compare the different projected in-situ instruc-tions in the context of impaired workers using an assistivesystem in manual manufacturing.

2

Figure 2. The assembly workplace is augmented with an assistive systemconsisting of a depth-camera and a projector (A). It enables displayingin-situ projected instructions during the working task directly onto theboxes holding the parts (B) and the working area (C).

SYSTEMFor exploring different visualizations of instructions in a man-ual assembly environment, we chose to augment an assemblyworkplace, i.e. a clamp-producing machine, with a projec-tor and a depth-camera. In the following, we introduce theprototype, the workflow, and the three visualizations.

Hardware Setup and WorkflowThe assembly workplace (see Figure 2) is an automated ma-chine that uses hydraulic and pneumatic systems to assemblea clamp from five parts. The parts have to be inserted at adefined position in the machine. The machine is extended witha Microsoft Kinect depth camera and a Casio AJ-X251 pro-jector. Both projector and depth camera are mounted on topof the machine to be able to project directly onto the workingarea and onto the boxes holding the parts. Furthermore, themachine was designed in a way that each part can be insertedfrom above and that both part and position of the part arealways visible for the camera and the projector.

The workflow for producing a clamp consists of picking andinserting five parts, closing a safety glass, and removing theglass once the machine finished assembling the clamp. Aspicking and placing a part are considered as two differentworking steps, the whole workflow consists of twelve stepsin total. We can differentiate between three types of steps:picking a part out of a box, inserting a part into the machine,and special actions i.e. closing the safety glass.

VisualizationsFrom related work, we identified and extended three typesof visualizations for projected instructions that can be under-stood by impaired workers. Additionally, from interviewswith supervisors of a sheltered work organization, we iden-tified that new workflows are normally taught to impairedworkers by a supervisor. Thereby, the supervisor lets the im-paired worker perform the workflow and simultaneously gives

verbal instructions. As the workers are used to get additionalverbal instructions, we used pre-defined verbal instructions inaddition to each projected feedback.

In spite of the fact that related work suggests textual instruc-tions for persons with mild cognitive disabilities, we could notconduct the experiment using textual instructions as many ofthe impaired workers were not able to read.

Pictorial InstructionsA way of presenting instructions are pictorial instructions.For picking parts from the boxes, we use a pictogram that isdisplayed directly in front of the box that the next part needs tobe picked from (see Figure 3a). The pictogram shows a handthat is picking something from a box and an arrow pointingaway from the box. For placing the part into the machine,we projected a full-sized picture of the part directly at theposition where it needs to be placed (see Figure 3d). Forclosing and opening the safety glass, we projected a picturenext to the glass, showing a hand that is moving the safetyglass and an arrow that indicates the direction (see Figure 3c).As the whole machine does not fit in the worker’s field of viewwhen standing in front of it, the projected pictures are slightlyblinking so that they can be better spotted.

Video InstructionsAnother way to visualize instructions is projecting a video ofthe action that needs to be performed. We chose to displayshort video clips that only show one action that the workerneeds to perform next. For picking parts from the boxes, avideo showing the picking of the part is shown directly underthe box to pick from. As the space for showing a video insidethe working area is too small, the system shows the video ofhow to place a part at the white projection area next to theworking area (see Figure 3e). The video shows where the partshould be placed in the machine. For both opening and closingthe safety glass, a different video is displayed at the projectionarea. Again, the video shows how to open and close the safetyglass.

Contour InstructionsPrevious work suggested that good instructions show the fea-tures of the parts that are changed in the current working step.Other features of the part that are not relevant for the stepshould be simplified. Therefore, we decided to provide a pic-torial instruction that just shows the contour of the currentpart. The contour transmits all relevant features, i.e. positionand orientation but hides potentially confusing details. In thisvisualization, for picking a part from a box the system justhighlights the box to pick from using a green light (see Figure3b). When placing a part, the system displays the full-sizedcontour of the part at the correct position also using a greenlight (see Figure 3f). Closing and opening the safety glass isindicated by displaying a green arrow in the direction the glassneeds to be moved next to it. As in the pictorial visualization,we designed the contours to be slightly blinking to enable theworkers to spot them easier.

EVALUATIONTo compare the three introduced visualizations of in-situ pro-jected instructions to a control group using no visual feedback,

3

(a) (b) (c)

(d) (e) (f)Figure 3. An overview of the different visualizations that were used in the study. For picking a part from a box, (a) shows a pictorial instruction and (b)shows highlights it as part of the contour visualization. (c) shows the image that was displayed when closing the safety glass. The position of the handleis visualized: (d) using a pictorial instruction, (e) using a video, and (f) using the contour.

Figure 4. An excerpt of the modified NASA-TLX questionnaire in En-glish.

we conducted a study with impaired workers as participants.As we want to assess the perceived cognitive load that a visu-alization induces and a regular NASA-TLX [8] would be toocomplex for a cognitively impaired person to file, we designeda simplified version of the NASA-TLX questionnaire that isspecially tailored to impaired participants.

A Simplified NASA-TLX for impaired participantsWe simplified the NASA-TLX [8] questionnaire by reducingthe number of choices for each of the six questions to fouroptions. The even number was chosen deliberately to avoidthe tendency towards the middle. According to Bortz & Doer-ing [3], one of the disadvantages of a typical five point or sevenpoint scale (Likert scale) is that the middle settings cannot beclearly interpreted. Participants may have misunderstood thecontent of the item, have felt mediocre,irrelevant, or have noopinion [6]. Therefore Bortz & Doering [3] recommend aninstruction how these values can be understood and suggest theuse of an even-numbered (e.g. four-stage ) scale. A four-stage

scale makes sense in terms of the described target group, asthey often tend to have no opinion [6].

Each of the four available choices is combined with a smi-ley indicating happiness when agreeing with the question orsadness when disagreeing with the question (see Figure 4).The smileys indicate to which extent the participants agreeor disagree with the question. Further, we put the text intoeasy language to be better understandable by impaired partici-pants.5

StudyFor evaluating the visualizations in a study with impairedworkers, we chose a between-subjects design with four groupsconsisting of the three visualizations and a control group usingno visual feedback. The only independent variable is thetype of the projected instruction. As dependent variables, wemeasure the task completion time (TCT), error rate (ER), andthe previously described simplified NASA-TLX for impairedparticipants.

As we wanted to have balanced groups for each visualization,we asked the supervisors of the impaired workers to assigneach worker to one of three categories according to his or herperformance index. The performance index (PI) is measuredin percent and indicates to which extent an impaired workeris capable of performing a task that a normal worker woulddo. The categories were defined by the supervisors of thesheltered work organization where we conducted the study.They were divided into groups as follows: PI of 5%-10%, PIof 15%-35%, and a PI over 40%. Each group has the samenumber of participants belonging to each PI group.5The full questionnaire can be downloaded from our website: http://www.motioneap.de/tlx-for-impaired-participants/

4

Figure 5. The task completion time in dependency of the groups usingdifferent visualizations. Error bars indicate the standard error.

ParticipantsWe recruited 64 participants (41 male, 23 female) for the study.The participants were aged from 16 to 59 years (M=41.7,SD=10.6). All participants were employees of a sheltered workorganization and were either mentally impaired or mentallyill. None of the participants were familiar with the clamp-producing machine. However, 67% of the participants hadexperiences in manual manufacturing. For each participant,the study took approximately 20 minutes.

ProcedureAfter welcoming the participant and explaining the course ofthe study, a general introduction about assistive systems forthe workplace was given. At the beginning, each participantwas assigned to one socio-educational instructor to supporthim or her during the task. This instructor stayed with theparticipant during the whole process to have a familiar personin this scenario, which was new to the impaired workers. Theinstructor then helped the impaired participant to complete aninitial questionnaire collecting demographic information andprior experiences in manufacturing. Afterwards, the partici-pant was assigned to one of the four conditions. To familiarizethe participant with the assembly of the clamp, the participantcould assemble it once using the chosen visualization of thefeedback while the instructor was giving verbal instructions.Afterwards, the participant had to assemble a clamp threetimes using only the visual feedback of the system accordingto the condition. The system was recording the time that wasneeded for each assembly task. The feedback was processedby a wizard-of-oz, who also counted the errors that were madeduring the assembly. After the assembly was finished, theparticipant was guided to a calm area where they filled in thesimplified NASA-TLX. At the end additional qualitative feed-back was collected. We randomly selected 13 participants totake part in a semi-structured interview, where we asked themabout satisfaction, motivation, self-reliance and challengescaused by the usage of the system.

ResultsOut of the 64 participants, two participants using the videocondition were not able to complete the study because theywere afraid of the projected videos. Thus, we excluded thetwo from the evaluation resulting in a total of 62 participants.

Figure 6. Error rate according to the visualizations used by the differentgroups. Error bars indicate the standard error.

We statistically compared the TCT, ER, and the simplifiedNASA-TLX between the groups using the different visualiza-tions. The assumption of homogeneity of variance had notbeen violated (p > .05) for the TCT. A one-way ANOVAtest revealed no statistical significant effect on TCT betweenthe groups (F(3,58)=1.446, p > .05). The effect size es-timate shows a medium effect (η2 = .069). The groupusing the contour visualization was the fastest (M=76.52s,SD=42.07s), followed by the group using the pictorial visual-ization (M=98.13s, SD=50.18s) and the control group usingno visual feedback (M=106.53s, SD=56.74s). The group us-ing the video visualization took the longest time to assemble(M=110.45s, SD=48.46s). An overview of the TCT in thevisualization groups is depicted in Figure 5.

For analyzing the ER between the groups, we used a non-parametric Kruskall-Wallis-ANOVA as there were indicationsthat the ER was not normally distributed. Again, the as-sumption of homogeneity of variance had not been violated(p > .05). The Kruskall-Wallis-ANOVA did not reveal asignificant difference χ2(3)=7.031, p = .071 > .05, but astatistical trend. The effect size estimate shows a mediumeffect (η2 = .11). The group using the contour visualizationmade the fewest errors (M=.93, SD=1.57), followed by thegroup using the video visualization (M=1.45, SD=1.75) andthe pictorial visualization (M=1.60, SD=1.82). The controlgroup using no visual feedback made the most errors (M=2.52,SD=2.07). An overview of the ER according to the differentvisualizations is depicted in Figure 6.

We statistically compared the results of the simplifiedNASA-TLX between the groups using a non-parametricKruskall-Wallis-ANOVA. As a post-hoc test, we used theWilcoxon signed-rank test with an applied Bonferroni correc-tion for all types of feedback, resulting in a significance levelof p < .0125. The simplified NASA-TLX could be filed byall 62 impaired participants taking part in the study.

Mental demand. Considering the mental demand, the testrevealed a significant difference χ2(3)=8.000, p = .046. Pair-wise Wilcoxon tests revealed that there is a significant differ-ence between the group without visual feedback and the groupusing contour visualization (Z = -2.572, p = .01). The otherpairwise tests did not reveal a significant difference (withoutvs. pictorial: Z = -1.442, p =n.s., without vs. video: Z =

5

Figure 7. Overview about the results of the modified NASA-TLX. Theerror bars indicate standard error. (A) Mental demand, (B) Physicaldemand, (C) Temporal demand, (D) Performance, (E) Effort, (F) Frus-tration.

-.303, p=n.s., pictorial vs. contour: Z = -1.033, p=n.s., picto-rial vs. video: Z = -1.117, p= n.s., contour vs. video: Z =-2.197, p = n.s.). The contour visualization was perceived theeasiest (M=.44, SD=.62), followed by the pictorial visualiza-tion (M=.81, SD=.98), and the video visualization (M=1.21,SD=1.05). The group using no visual feedback perceived thetask most complex (M=1.31, SD=1.01) (see Figure 7 (A)).

Physical demand. Regarding the physical demand, the testcould not find a significant difference between the groups us-ing the different visualizations χ2(3)=4.278, p = n.s.. Theparticipants perceived the contour visualization as the leastphysically demanding feedback (M=.44, SD=.62), followedby the pictorial visualization (M=.81, SD=.98), and the videovisualization (M=1.21, SD=1.05). The group using no vi-sual feedback reported the highest physical demand (M=1,31,SD=1.01) (see Figure 7 (B)).

Temporal demand. The analysis of the perceived temporal de-mand did not reveal a significant difference between the groupsusing the different visualizations χ2(3)=3.161, p = n.s.. AsFigure 7 (C) shows, the contour visualization was perceivedthe fastest (M=.50, SD=.73), followed by the pictorial visual-ization (M=.75, SD=.77), and without visualization (M=.87,SD=.71). The video visualization was perceived the mosttemporal demanding (M=.93, SD=.82).

Performance. We statistically compared the perceived perfor-mance of the participants between the different visualizationgroups. The test revealed a significant difference betweenthe groups χ2(3)=8.493, p = .037. Pairwise Wilcoxon testsrevealed a significant difference between the group withoutvisual feedback and the group using contour feedback (Z =-2.575, p = .01). The other pairwise tests did not reveal asignificant difference (without vs. video: Z= -.695, p=n.s.,pictorial vs. video: Z= -.794, p=n.s., contour vs. video: Z=-2.433, p = n.s., pictorial vs. contour: Z= -1,506, p=n.s.,and without vs. pictorial: Z= -1,287, p=n.s.). Participantsperceived their performance best using the contour visual-ization(M=.38, SD=.80), followed by the pictorial visualiza-tion (M=.81, SD=.98), and the video visualization (M=1.00,SD=.78). Participants using no visual feedback perceivedtheir performance as least successful (M=1.31, SD=1,13). Anoverview is depicted in Figure 7 (D).

Effort. The statistical comparison of the participant’s perceivedeffort between the different visualization groups did not reveala significant difference χ2(3)=1.427, p = n.s.. However, theparticipants perceived the lowest effort using the contour visu-alization (M=.63, SD=.80), followed by the pictorial visual-ization (M=.75, SD=.77), and the video visualization (M=.86,SD=.94). As depicted in Figure 7 (E), the group using novisual feedback perceived their effort the highest (M=1.00,SD=.96).

Frustration. Finally, we compare the frustration that was per-ceived by the participants using the different visualization.The test revealed a significant difference between the groupsχ2(3)=8.149, p = .043. However, pairwise Wilcoxon testsdid not reveal any significant difference between the differentvisualizations (without vs. video: Z= -.183, p=n.s., pictorial vs.video: Z= -1.569, p=n.s., contour vs. video: Z= -2.388, p=n.s.,without vs. contour: Z= -2.334, p=n.s., pictorial vs. contour:Z= -.821, p=n.s., without vs. pictorial: Z= -1.467, p=n.s.).Participants found the contour visualization the least frustrat-ing (M=.25, SD=.57), followed by the pictorial visualization(M=.44, SD=.72), and the group using no visual feedback(M=.75, SD=.68). As depicted in Figure 7 (F), the group usingthe video feedback reported the highest frustration (M=.86,SD=.86).

We further analyzed the participant’s performance accordingto the PI groups using a one-way ANOVA. Regarding the ER,we found a significant difference (F(2,59)=7.251, p = 0.002).A Tukey HSD post-hoc test revealed a significant differencebetween PI 40%+ and PI 5%-10%, as well as between PI 40+and PI 15%-35%. Regarding the TCT, the test also founda significant difference (F(2,59)=7.999, p = 0.001). Again,a Tukey HSD post-hoc test revealed a significant differencebetween PI 40%+ and PI 5%-10%, as well as between PI 40%+and PI 15%-35%.

Finally, we compared the effect between the different visu-alizations and the PI groups regarding the TCT. A two-wayANOVA revealed a significant effect between the PI groups forthe group using no visualization (F(2,50)=4.878, p = 0.012)and the group using the pictorial feedback (F(2,50)=5.590,p = 0.006). Pairwise comparisons reveal a significant dif-ference between PI 5%-10% and PI 40%+, and between PI5%-10% and PI 15%-35% for the group using no visual feed-back. Regarding the pictorial feedback group, it only revealeda significant difference between PI 5%-10% and PI 40%+.

We furthermore compared the effect between the different vi-sualizations and the PI groups regarding the ER. A two-wayANOVA revealed a significant effect between the PI groups forthe group using the pictorial visualization (F(2,50)=3.455, p =0.039) and the group using no visual feedback (F(2,50)=5.996,p = 0.005). Pairwise comparisons reveal a significant differ-ence between PI 5%-10% and PI 40%+ for the group usingthe pictorial visualization. Regarding the group using no vi-sual feedback, the comparison reveal a significant differencebetween PI 5%-10% and PI 40%+, and between PI 5%-10%and PI 15%-35%.

6

Figure 8. The task completion time according to the used visualizationand the Performance Index (PI) of the participants. The contour visual-ization especially helps participants with the lowest PI.

The qualitative feedback revealed that the contour visualiza-tion was well perceived by the participants. P14 stated thathe ”could see the alignment of the part from the shape of theprojection”. Regarding the video visualization, we observedthat the frustration of the participants using the video wasextremely high. We even needed to abort two experimentswith the video visualization because the two participants werescared of the videos and started panicking. We could alsoobserve that most participants using the video condition werenot watching the video fully. They were just looking at it occa-sionally. P37 stated that he ”did not understand that the videotells me what to do”. The participant inferred the position ofthe part based on the affordance.

In the semi-structured interview, we randomly selected 13 par-ticipants from different PI groups and different visualizationgroups. When being asked about the general idea of visualfeedback during the working process, 12 of the 13 interviewedparticipants stated that they felt adequately supported by thesystem. A total of 12 of the 13 participants experienced joywhile working on the system, which leads to increased motiva-tion at work. All participants feel an increased independencyand self-responsibility in their work task, by using the system.E.g. P11 states that she ”is able to work confidently becauseof the flashing lights”, and P5 states that he”got help by theinstructions and the lights of the system”. Only 3 of the 13interviewed participants feel challenged by working at thesystem. These participants expressed difficulties in learningand comprehending the instructions.

DISCUSSIONThe results of the study reveal that the contour visualizationinduces the fewest errors and the shortest TCT. Although thedifference in committed errors and TCT is not statisticallysignificant, we could observe a trend favoring the contourvisualization. Regarding the TCT, the video visualizationeven induced a longer TCT than the control group using novisual feedback. However, the difference is not statisticallysignificant. A reason for this trend could be the additional timeto view the video.

On the other hand, the results show that the contour visualiza-tion induces significantly less perceived mental demand thanthe control group using no visual feedback. Furthermore, theperceived performance of the participants was significantly

Figure 9. The error rate according to the used visualization and thePerformance Index (PI) groups. The contour visualization causes thefewest errors for all PI groups.

better using the contour visualization compared to the con-trol group using no visual feedback. Overall, the contourvisualization performed best in all six measures of the simpli-fied NASA-TLX, however no significant differences could befound in the other four measures.

Considering the different PI groups, the results revealed astatistically significant difference regarding the TCT and theER for the pictorial feedback and the control condition withoutvisual feedback. For the video and the contour feedback, nostatistically significant difference was found. This indicatesthat the contour feedback and the video feedback caused thePI groups to achieve results of similar quality (see Figure 8and Figure 9). This is especially remarkable for the PI group5%-10% which using contour feedback achieves TCTs andERs comparable to the other groups. Thus, contour feedbackmight provide a means to enable more impaired workers toimprove their performance.

Finally, visual support was generally well perceived by the par-ticipants. Many participants felt more confident when viewingprojected instructions directly at the working place.

CONCLUSIONIn this paper, we presented a simple contour-based visual-ization as projected in-situ feedback for displaying workinginstructions at an assistive system for the workplace that caneasily be understood by impaired workers. Through a userstudy involving 64 impaired participants, we compared thecontour visualization, a pictorial visualization, and a videovisualization in a real world working scenario, i.e. assemblinga clamp. Additionally, we introduce a simplified version ofthe NASA-Task Load Index questionnaire, that was specifi-cally designed to be understood by impaired participants. Theresults of the study reveal, that the perceived performanceis significantly higher and the perceived mental load is sig-nificantly lower using the contour visualization compared tousing no visual feedback. Further, the results indicate thatthe contour visualization performs best concerning error rateand task completion time, however not statistically significant.The participants generally liked the visual assistance providedby our prototype and would like to use visual feedback at theirworking place every day.

As future work, the top-mounted depth-camera will be usedto automatically detect performed actions and to proceed the

7

feedback automatically. Thus, an automatic assistive systemthat supports impaired workers in real-time during their dailyworking tasks becomes a technical possibility.

ACKNOWLEDGMENTSThis work is funded by the German Federal Ministry for Eco-nomic Affairs and Energy in the project motionEAP, grant no.01MT12021E. We thank the Gemeinnutzige Werkstatten undWohnstatten GmbH for supporting us. We especially wouldlike to thank Frank Raschhofer for organizing the study.

REFERENCES1. Agrawala, M., Phan, D., Heiser, J., Haymaker, J.,

Klingner, J., Hanrahan, P., and Tversky, B. Designingeffective step-by-step assembly instructions. In ACMTransactions on Graphics (TOG), vol. 22, ACM (2003),828–837.

2. Bannat, A., Wallhoff, F., Rigoll, G., Friesdorf, F., Bubb,H., Stork, S., Muller, H., Schubo, A., Wiesbeck, M., andZah, M. Towards optimal worker assistance: a frameworkfor adaptive selection and presentation of assemblyinstructions. In Proceedings of the 1st internationalworkshop on cognition for technical systems, Cotesys(2008).

3. Bortz, J., and Doring, N. Forschungsmethoden undevaluation: fur human-und sozialwissenschaftler.Springer, 2006.

4. Butz, A., Schneider, M., and Spassova, M. Searchlight–alightweight search function for pervasive environments.In Pervasive Computing. Springer, 2004, 351–356.

5. Cuvo, A. J., Davis, P. K., O’Reilly, M. F., Mooney, B. M.,and Crowley, R. Promoting stimulus control with textualprompts and performance feedback for persons with milddisabilities. Journal of applied behavior analysis 25, 2(1992), 477–489.

6. Goeke, S., and Bundesvereinigung Lebenshilfe furMenschen mit Geistiger Behinderung. Frauen starkensich: Empowermentprozesse von Frauen mitBehinderungserfahrung; eine Studie im Stil der groundedtheory methodology. Lebenshilfe-Verlag, 2010.

7. Hardy, J., and Alexander, J. Toolkit support forinteractive projected displays. In Proceedings of the 11thInternational Conference on Mobile and UbiquitousMultimedia, ACM (2012), 42.

8. Hart, S. G., and Staveland, L. E. Development of nasa-tlx(task load index): Results of empirical and theoreticalresearch. Advances in psychology 52 (1988), 139–183.

9. Heiser, J., Phan, D., Agrawala, M., Tversky, B., andHanrahan, P. Identification and validation of cognitivedesign principles for automated generation of assemblyinstructions. In Proceedings of the working conference onAdvanced Visual Interfaces, ACM (2004), 311–319.

10. Hendricks, A. Un convention on the rights of personswith disabilities. Eur. J. Health L. 14 (2007), 273.

11. Johnson, B. F., and Cuvo, A. J. Teaching mentallyretarded adults to cook. Behavior Modification 5, 2(1981), 187–202.

12. Korn, O., Funk, M., and Schmidt, A. Design approachesfor the gamification of production environments. a studyfocusing on acceptance. In Proceedings of the 8thInternational Conference on PErvasive TechnologiesRelated to Assistive Environments, ACM (2015).

13. Korn, O., Schmidt, A., and Horz, T. Augmentedmanufacturing: a study with impaired persons on assistivesystems using in-situ projection. In Proceedings of the6th International Conference on PErvasive TechnologiesRelated to Assistive Environments, ACM (2013), 21.

14. Lancioni, G. E., O’Reilly, M. F., Seedhouse, P., Furniss,F., and Cunha, B. Promoting independent taskperformance by persons with severe developmentaldisabilities through a new computer-aided system.Behavior Modification 24, 5 (2000), 700–718.

15. Linder, N., and Maes, P. Luminar: portable roboticaugmented reality interface design and prototype. InAdjunct proceedings of the 23nd annual ACM symposiumon User interface software and technology, ACM (2010),395–396.

16. Pierce, K. L., and Schreibman, L. Teaching daily livingskills to children with autism in unsupervised settingsthrough pictorial self-management. Journal of appliedbehavior analysis 27, 3 (1994), 471–481.

17. Pinhanez, C. The everywhere displays projector: Adevice to create ubiquitous graphical interfaces. InUbicomp 2001: Ubiquitous Computing, Springer (2001),315–331.

18. Ruther, S., Hermann, T., Mracek, M., Kopp, S., and Steil,J. An assistance system for guiding workers in centralsterilization supply departments. In Proceedings of the6th International Conference on PErvasive TechnologiesRelated to Assistive Environments, ACM (2013), 3.

19. Sauer, A. L., Parks, A., and Heyn, P. C. Assistivetechnology effects on the employment outcomes forpeople with cognitive disabilities: a systematic review.Disability & Rehabilitation: Assistive Technology 5, 6(2010), 377–391.

20. Schuntermann, M. Einfuhrung in die icf: Grundkurs,ubungen, offene fragen. 2. Aufl. Landsberg/Lech: EcomedMedizin (2007).

21. Steed, S. E., and Lutzker, J. R. Using picture prompts toteach an adult with developmental disabilities toindependently complete vocational tasks. Journal ofDevelopmental and Physical Disabilities 9, 2 (1997),117–133.

22. Wilson, A. D. Using a depth camera as a touch sensor. InACM international conference on interactive tabletopsand surfaces, ACM (2010), 69–72.

23. Zhou, J., Lee, I., Thomas, B., Menassa, R., Farrant, A.,and Sansome, A. Applying spatial augmented reality tofacilitate in-situ support for automotive spot weldinginspection. In Proceedings of the 10th InternationalConference on Virtual Reality Continuum and ItsApplications in Industry, ACM (2011), 195–200.

8