Towards an Intuitive Sign Language Animation AuthoringEnvironment For the Deaf

Alexis HeloirUVHC, LAMIH, F-59313 Valenciennes, FranceDFKI-MMCI, SLSI group, D-66123, Germany

alexis.heloir@dfki.de

Fabrizio NunnariDFKI-MMCI, SLSI group, D-66123, Germany

fabrizio.nunnari@dfki.de

ABSTRACTWe are aiming at developing an online collaborative frame-work allowing Deaf individuals to author intelligible signsusing a dedicated authoring interface controlling the anima-tion of a 3D avatar. In this paper, we present the results thatwe have achieved so far. They mainly focus on the designof a User Interface assisted by novel input devices. In par-ticular, we show, in two user studies, how the Leap Motionand Kinect-like devices can be used together for capturingrespectively hand trajectories (position and orientation) andfacial animation. We also show how these devices are inte-grated into a prototype that we will demonstrate during theworkshop.

Keywordsavatars, animation authoring, natural user interface, inter-action

1. INTRODUCTIONDeaf individuals often acquire a sign language as their firstlanguage and are most fluent and comfortable in this firstlanguage. For these individuals, sign language interfacesare highly desirable. Despite the fact that many deaf indi-viduals are skilled readers, not all deaf signers develop thislevel of proficiency. The reasons may be varied, but thisphenomenon is replicated worldwide, regardless of sign lan-guage or written language of the country. This literacy issuehas become more significant in recent decades, as new infor-mation and communications technologies have arisen thatplace an even greater premium on written language literacyin modern society. For spoken languages, a computer systemcan display written text onto the screen for the user. Forsign languages, this approach is generally not possible. Thecoordinated use of multiple parts of a signer’s body during asign language performance and the use of 3D space aroundthe signer is challenging to encode in a written representa-tion.

So far, no sign language written representation system hasbeen adopted for sign language: without a community ofusers that accept and have developed literacy skills in oneof these systems, none can be used as output on a sign lan-guage interface. Therefore, the output must be displayed inthe form of a video or animation of a human-like charactersigning. Animated sign language avatars on the Internet oron mobile devices have the potential to significantly improvethe situation for the Deaf. However, producing comprehen-sible sign language avatars remains a difficult problem.

We are developing an online collaborative framework thatwould allow Deaf individuals to author intelligible signs us-ing a dedicated authoring interface controlling the animationof a 3D avatar. This tool would enable deaf individuals fromdifferent linguistic communities to create their own anima-tions in order to illustrate new concepts, invent new signsand populate dictionaries. Eventually, it might be an alter-native to video recording, unlocking anonymous expressionfor deaf individuals on the Internet using their primary lan-guage. Such a tool would also put Sign Language studiesback into the hands of the Deaf by allowing them to pop-ulate large corpus of animation data – potentially a veryvaluable research material for the advancement of Sign Lan-guage linguistics.

Our work is by nature pluridisciplinary and rely on User In-terfaces, innovative input devices, collaborative research. Inthis paper, we present intermediate results that have beenachieved so far. They mainly focus on User Interface de-sign and exploit new input device. In particular, we showhow the Leap Motion device and the Kinect can be use tocapture respectively hand trajectories (position and orienta-tion) and facial animation. We then show how these devicesare integrated into a prototype that we might demonstrateduring the workshop.

The rest of the paper is organized as follows: next Sectionpresents and discuss the related work, Section 3 presents anoverview of our architecture, Section 4 presents our latestachievements using the Leap Motion and the Kinect devicefor hand trajectory and facial animation, Section 6 concludesthe paper.

2. RELATED WORKSigning avatars are a relatively young research area withtwo decades of active research. Since the early attempts ofLebourque and Losson [21, 15], directly inspired from the

articulatory representation of Stokoe [32], two influencialEuropean projects, ViSiCAST and eSIGN, developed tech-nology for signing avatars based on HamNoSys [8, 12], tran-sitioning from a concatenative to an articulatory approach,and advancing HamNoSys to SiGML. A more recent avatarproject is called Paula with a number of interesting resultsfor the synthesis of fingerspelling, nonverbal components andnatural pose computation [36]. This project relies mostly onmanually crafted sign-language annimation, like in Delormeet al. [6] or in the ATLAS project [20], where a “neutral”version of each sign is stored as a complex full-body + facialanimation. During sentence synthesis, signs are adaptedto the context of the sentence by mixing the neutral signwith specific facial expressions (e.g., eyebrows down), bodypostures (e.g., torso leaning forward), or parametrized pro-cedures (e.g., relocation of sign in space). These projects,however work at at GLOSS level and can be considered asconcatenative approaches, even if they propose some tem-platization and on-the-fly adaptation of the animated signs.

In our own research, we recently evaluated in tight collab-oration with deaf experts a set of manually authored ani-mations. for the first time, we assessed the intelligibility ofour animations using complex sentences that we comparedagainst reference recordings performed by professional deafinstructors. For this study, the animation tool was devel-oped as a research prototype and could only be used by afew people in the lab, a deaf expect provided instructions onhow to author the sign language animations. Interestingly,Deaf instructors always preferred animations that were lessnatural but more articulated and intelligible. We could alsoshow that acceptance towards Deaf community could be sig-nificantly increased by involving deaf participants [13]. In-volving deaf individuals in the research effort requires dedi-cated tools that are at the same time easy to use and havethe potential to author complex multi-channel animations.

To our knowledge, the closest attempt to the editing plat-form we propose has been presented by Jemni et al. in2008 [11]. In this work, deaf users are supposed to au-thor signs using a VRML/H-ANIM character driven by aHTML/ActiveX GUI. Unfortunately we couldn’t find a plat-form supporting the requirements of their online demonstra-tor1. In this work, interactions are only conducted usingkeyboard and mouse, contrary to our apporach which usesNatural Input Devices (NID) like the Leap Motion and theMicrosoft Kinect. More recently, Adamo-Villani et al. [1]introduced an easy to use system specifically focusing onsign language animation. This system allows users to posesimple avatars rendered in the 3D viewport using special-ized 2D interfaces displayed in a separate 2D window us-ing keyboard and mouse. Authors claim that 2D interfacesare more suited to novice users. While we agree on theimportance of providing the novice user with simple inter-faces and a minimal number of fixed-viewport 3D renderwindows, we also believe that, when it comes to positioningobjects or end-effectors in space, a 3D input device providinga direct mapping between the user’s hand and the position-ing/orientation of manipulated 3D objects in a fixed viewmight be an interesting complement to the specialized 2Dinterfaces proposed by Adamo-Villani et al.

1http://hebergcck224.rnu.tn/ws/ site/demo.php, July 13th

Recently, a number of work tried to leverage the potential ofCrowd-sourcing approaches for populating dictionaries withnew signs and concepts. For instance, Cavender et al. pro-pose a collaborative platform supporting the creation of newsigns in the STEM (Science, Technology, Engineering, andMathematics) disciplines. One could also cite Culinan2 andElix3 for French Sign Language. These attempts, howeverrelied exclusively on video recording. We claim that videorecordings, because they do not carry any symbolic repre-sentation of the signs they depict are not as interesting asanimations described in a symbolic language like, for in-stance EMBRScript [9]. Pengfei et al. [22], use cutting-edgemotion capture technology to record large corpus of Deafindividuals telling stories and interacting. The motion datathey retrieve is animated, in 3D and extremely refined. It istherefore a great material for animators and linguists. Webelieve that our approach is complementary and provide in-formation that is reflects better the author’s actual intent:we could for instance, monitor, record and analyse the be-havior of Deaf users using our authoring tool to extract spe-cific pattern and find out important features that make asign intelligible, according to the Deaf.

Animation is a highly cross-disciplinary domain that spansacross acting, psychology, movie-making, computer anima-tion, and programming. Not surprisingly, learning the artof traditional animation requires time, effort and dedica-tion. However, recent consumer-range technology [28] hasproved to be capable of enabling inexperienced users author-ing animation of human-like bodies or interactively control-ling physical or digital puppets. We are aiming at a simi-lar goal for sign language animation. Our system is inno-vative because not only it allows novice users to naturallyedit complex animations using natural input devices thatare the Kinect and the Leap motion, it also allows them toswitch seamlessly between traditional space-time constraintedit and interactive performance capture recording. Indeed,some devices are better suited for live-capturing the dynam-ics of the animation [33, 5] while other devices are moresuited to off-line single pose edit [1, 17]. Only few systemare actually suited to both methods [14] and to our knowl-edge, no proposed architecture accounts for the possibility toswitch seamlessly between the two modes during an editingsession. Addressing this issue and empowering the anima-tion workflow with natural interaction metaphors are twocontributions of the system we propose.

2.1 Pose-to-pose animationPosing an articulated human figure consists into setting-upthe many controls composing its kinematic structure in orderto satisfy a higher level specification. In traditional anima-tion, these controls are exposed by control structure calledthe rig. Creating a rig consists of setting up a group of con-trols that operate a 3D model, analogous to the strings of apuppet. It plays a fundamental role in the animation pro-cess as it eases the manipulation and editing of expressions.It is essential for the animator to have a rig that allows aright expression palette, that is convenient to use, and at thesame time compact (not too many handles) and expressive(the handles provide a good factorization of the controllers).

2http://www.culinan.net/3http://www.elix-lsf.fr/

The Blender software’s UI that we use in this work [27],like other UI approaches [16, 30] represent the rig-handleswith in the 3D viewport by using direct manipulation in-terfaces. Driving a complex structure with low-dimensionalcontrol signals has been addressed by Chai et al. [5]. Ourdimension-reduction technique is not data driven but relyon classical inverse kinematics. We also do not use a tangi-ble object as an input device, but an infrared depth-cameraprovided by the Leap Motion device. In the pose-to-poseedit scheme, our system uses the Leap motion to drive thearm/shoulder complex following a relative drag and releasecontrol of a character’s.

2.2 Performance captureIn 1988, Robertson demoed the first interactive anthropo-morphic computer puppet: Mike [26]. Mike was capable ofperforming simple facial expressions and moving the lips ac-cording to the puppeteer’s speech. Mike’s animation rig con-sisted of data glove and a speech recognition system. Mikewas later followed by systems capable of capturing in realtime the animation of an actor’s face without markers [35, 4].When it includes face, Motion Capture [23] is often referredto as performance capture. In performance capture, the pup-peteer’s face usually does not match the puppet’s face: theyoften have different topology and morphology. The map-ping of the puppeteer’s facial motion on the puppet face iscalled retargetting [31, 18, 25]. In the work we present, weuse the technology developed by Weise et al. [35] with adedicated facial motion retargeting stage which, followingthe terms introduced by Pighin [25], is based on a scattereddata interpolation approach (the mapping is driven by a ma-trix multiplication function) fed by an art directed input (wemanually edited the correspondences (matrix rows) betweeneach source blend shape to a target control rig configura-tion).

Neff et al. [24] suggest an approach for mapping 2D mouseinput to high-dimensional skeleton space with so-called cor-relation maps that are learned on motion capture data. How-ever, their approach has been proved to be effective onlyfor postural expression animations like dancing. Compara-ble approach has been adopted in Motion Doodles [33], byThorne et al. who presented a system for sketching the mo-tion of a character using 2D trajectories. The trajectoriesare parsed and mapped to a parameterized set of outputmotions that further reflect the location and timing of theinput sketch. The system uses a cursive motion specifica-tion that allows for fast experimentation and is easy to usefor non-experts. However, this approach does not attain theprecision required for co-verbal gesture or sign-language.

Kipp and Nguyen [14] proposed a multi-touch interface al-lowing an animator to interactively control an IK-driven hu-man arm and record poses or captured motions. They couldshow that the system can be easily learned by novice anima-tors and that the 2D multitouch interface they proposed en-ables naive users to produce highly coordinated hand anima-tions with subtle timings. The proposed interface could beused both for performance capture and pose-editing thoughthe proposed system considered the two methods indepen-dently. Lockwood et al. [19] used a touch-sensitive laptopas an input-device of a walk pattern generator controlled bytwo fingers walking on the laptop’s surface while Sanna et

al. [28] achieved a similar task for the whole body using theMicrosoft Kinect depth-camera. Our system achieve sim-ilar performance-capture task in the performance capturemode. It is however capable of seamlessly transitioning tothe pose-to-pose edit mode, enabling the animator to refineand improve its recorded performance by editing a limitednumber of inferred key-poses.

2.3 Take AwayTo sum up, the solutions presented in the Related Work sec-tion are either novel input devices that replace or completethe keyboard/mouse input for pose-to-pose edit or novel per-formance capture solutions supporting the straight-aheadauthoring scheme. Only one solution presented in the Re-lated Work Section could be considered both for editing ani-mation according to both straight-ahead or pose-to pose: theone presented by Kipp et al. [14]. In our work, we stress theimportance of setting up new interaction metaphors that areseamlessly integrated and that account for both authoringschemes. The interaction metaphors we design and evaluateare constructed upon the Leap Motion device. Our designchoices and system architecture are presented in the follow-ing paragraphs

3. PROPOSED DESIGNThe contribution of the architecture we propose is that itendows the user with the capability to seamlessly record andedit character animation using both pose-to-pose animationor performance capture. We propose a consistent frameworkdepicted in Fig. 1.

In performance capture mode, the animator drives the ani-mation like a puppeteer. The motion of his both of hands orface is tracked in real time and his live performance drivesthe animation of the avatar that is being recorded at a rateof 25 frames per seconds. The frame density is later reducedin order to allow a latter manual edit that is also conductedusing the Leap Motion. On the contrary, in the pose-editmode, the user controls one hand at a time using the LeapMotion. In this mode, the user hand posture is not directlyfollowed by the system, rather, it integrates sequences ofconsecutive relative edits that consists into small grab andrelease actions on one of the character’s hand. This editmode permits a much finer and more precise positioning ofthe character’s hand.

3.1 Performance capture modeTwo input devices are using simultaneously during perfor-mance capture: the Leap motion for the palms and theKinect for the face.

3.1.1 Using the Leap MotionPerformance-driven animation and puppeteering usually relyon a specialized input device or a motion-capture system.In performance capture (puppetry), the performer has adirect control over the virtual character movements. Theuser/animator has thus a sense of presence. That is accen-tuated by the mirrored display of the character: the user isable to see the hands of the animated character as it wouldsee his own hands in a mirror.

In order to give a user immediate feedback on how its cap-

Kinect

PerformanceCapture

PerformanceCapture

DataSimplification

DataSimplification

ManualEdit

ManualEdit

Cont

rol R

igCo

ntro

l Rig

Leap Motion

User

Straight-ahead

Pose-to-pose

AnimationAnimation

Figure 1: The authoring pipeline overview.

ture action in proceeding, the resulting animation appliedon the digital character should be immediately visible onthe screen, in real-time.

Figure 2: The mirror view used during performancecapture.

Performance capture poses several challenges: first, captur-ing hardware and software must be calibrated according tothe animator’s morphology before the actual capture. More-over, the morphology of the performer is hardly the same ofthe virtual character, animation data must be retargeted inreal time to the virtual character. Our implementation iscurrently tailored to one morphology and we are workingon a generic and robust calibration step that might fit thelargest range of users possible.

3.1.2 Using the KinectFaceShift is an application able to use a Kinect-like 3D depthsensor to reconstruct the 3D mesh of a face an animate it inreal-time according to the actual facial expression of a per-former. The details of its underlying implementation can beseen in [35]. Faceshift is based on the OpenNI4 middleware,guaranteeing compatibility with a wide range of Kinect-likehardware available on the consumer market. FaceShift re-quires a calibration phase for each user. The calibration,

4http://www.openni.org/ (28 July 2013)

lasting less than 10 minutes, consists in mimic a set of facialexpressions; while keeping an expression the user turns hishead of about 30 degrees on both sides, a couple of times, tolet the system collect data about the shape of the face. Af-ter the calibration FaceShift can be use in real-time trackingmode. Here the user face expression framed by the Kinectis analyzed and used to pilot a 3D reconstruction of the userface.

In order to animate and record in real-time the characterhead and face inside the Blender animation software, wedeveloped a python script which decodes the FaceShift net-work communication protocol and maps the received valuesinto control values for the Blender Software5 . We remindthat, in so doing, the Blender user can have access to theanimation keyframes to further edit them. The script givesthe possibility to record animation sessions or to imprint asingle keyframes on the timeline in order to freeze a sequenceof static facial expression and leave the interpolation workto Blender.

The FaceShift data, received through a UDP socket, whichwe used for our integration is the following: 1, a track ok flagindicating whether the user face is being correctly tracked ornot; 2, head rotation (yaw/pitch/roll angles as quaternion);3, eyes rotation (yaw and pitch angles for each eye); 4, a setof 48 float values, in the range [0,1], representing the weightof the 48 blend shapes associated with the head 3D model.

After recording, the resulting animation displays 25 recordedkey-frames per seconds, which are too many for a later man-ual edit. We therefore need a method that reduces the num-ber of keyframes after a performance has been captured.

3.2 Data simplificationIn our pipeline we sample the motion of the rig controllers,and not directly the underlying skeleton. This brings thenumber of DOFs to edit to a more human manageable con-dition, yet leaving the problem of an excessive sampling fre-quency (25 keyframes per second).

5http://www.blender.org/ (28 July 2013)

Hence, before being submitted to a human author, the ani-mation data must pass through a Data Simplification stage.The aim of this stage is to reduce the quantity of samplesof each DOF, making it reasonably simple to be edited bya human operator. This phase consist in procedurally ex-tract salient poses from an animation, like in [10]. A side-effect of this procedure is the elimination of the “noise” (i.e.the high frequency signals) of the performance capture, thatconveys the kinematic signature of the animator. This mightbe considered as a desired feature if we aim at providinganonymized gesture sequences.

An implementation of our simplification routine has beenpublished online6 and has been submitted to become partof the standard Blender7 distribution.

3.3 Pose-to-pose edit schemeDifferently from the first person view approach presentedpreviously, the user manipulates here the virtual characteras a puppet, adjusting the details of each key pose extractedfrom the previous phase and synchronising the recorded an-imation layers.

The purpose of the user is to use its own body to applyoffsets to the key poses resulting form the Data Simplifica-tion phase. The idea is to keep a correspondence betweenthe body parts of the author and the ones of the virtualcharacter. However, in contrast with the direct control of aperformance capture, here the author performs a “relative”control on the character current posture. For example, themovement of an author hand from an arbitrary position isused to apply an offset to the position of a hand of the vir-tual character. We can call this a form of body-coincidentpuppeteering.

Here the point of view on the digital character is external:the screen is not anymore a mirror, rather a window (ormore) on the 3D virtual world. Concerning the camera con-trol, we aim at an authoring setup where the user focus onlyon character posture editing without the need of controllingthe camera position; past studies already demonstrated thatseveral tricks can be applied to successfully enable depthperception in character animation, thus eliminating the needof rotating the camera viewpoint [14]. A detailed descrip-tion of the configuration used in the experiment reported inthis paper is described later.

4. EXPERIMENTS AND RESULTSBoth input methods – Leap Motion and Kinect Faceshifthave been tested independently. Both experiments havebeen conducted on a Mac Book Pro Laptop (2.4 GHz In-tel Core i7 CPU, 16GB Ram, OS X 10.8.4) connected to a22 inches monitor (resolution 1680x1050) at about 60 cm ofdistance from the eyes. For both studies, an operator wassitting next to the subject, monitoring the advancement ofthe experiment, switching between tasks and (de)activationthe logging system. We used the character and rig providedby MakeHuman v1.0 alpha7. Blender version was 2.66.1.

6http://slsi.dfki.de/software-and-resources/keyframe-reduction/ (28 July 2013)7http://www.blender.org (28 July 2013)

4.1 Evaluating the Leap MotionOur experiment focuses on positioning objects in 3D spaceand posing humanoid characters: these are both commontasks for animators. In theory, users could simultaneouslymove and rotate objects in the 3D space unsing th LeapMotion, thereby increase performance. We expect direct 3Dmanipulation to perform better than the mouse and key-board, at least for 3D object positioning. For single tar-get selection, Sears and Shneiderman [29] have shown thatdirect-touch outperforms the mouse.

4.1.1 Task and Experiment designThe evaluation has been carried out on two contexts. Thefirst consisted in positioning a 3D brick (position and orien-tation) in a 3D environment. The second consisted in posinga humanoid figure in the 3D environment using the handlesand the inverse kinematics provided by its animation rig.We compared the performance across two input conditions:1) Mouse and Keyboard (M&K) and 2) 3D Natural UserInterface (NUI) input. This comparison, however, can onlybe performed on subjects who already have experience with3D software. Since we couldn’t find a novice subject capa-ble of performing the 3D manipulation task using keyboardand mouse, in our experiment novice subjects only used theNUI. We could however compare the performance of novicesubjects using the 3D input versus the performance of ex-perienced subjects using the keyboard and mouse or the 3Dinput.

4.1.2 Subjects and ApparatusWe conducted the study on two subjects. The first subjectwas an expert Blender subject. He accomplished the taskswith both traditional Mouse and Keyboard (M&K) inputsystem and with the Natural Subject Interface (NUI). Asecond subject was a complete novice in 3D graphics. Heconducted the study using only the NUI approach since wewas not capable to accomplish the task using (M&K).

Figure 3 shows the screen setup we adopted for the exper-iment. The Blender editor was shown at full screen resolu-tion. The upper part of the editor was showing a frontalview of the context to edit (brick or character). At its sidea control panel was provided to switch between scenes andtasks. The bottom part of the editor was showing two copiesof the top view of the scene. The top view was provided ascue to better evaluate objects distance from the camera. Weprovided two copies of the top-view to ensure its visibilityalso when editing the scene while moving a hand in front ofthe screen.

4.1.3 ResultsFor each trial, we recorded the time spent by the subjectwhile manipulating the interface (hitting one of the G,T,Ror F key). We started the timer immediately after the sub-ject touched the first edit key to begin a trail (task) andstopped the timer as soon as the task was done, fulfilled,performed. We distinguished between the time spend whilemoving objects in the scene (i.e. re-locating and/or rotatingan object) from the time spent in switching between differ-ent editing modes.

Figure 4 shows the average time each participant took toaccomplish each scene. Results clearly show that the expert

Figure 3: The Blender layout used during the tasks.

subject was, in average two times faster with the NUI. More-over the novice subject was able (with the NUI) to matchthe editing speed of the expert using M&K.

Figure 4: Average time (seconds) needed to accom-plish the tasks.

4.2 Evaluating the KinectFor the face performance capture, we conducted the userevaluation in two stages. Firstly, 16 users volunteered touse the system at the lab and record a set of dynamic ex-pressions. Secondly, we set up an publicly advertised on-line questionnaire displaying the reference expression videos,video recordings of the users and rendered animations of therecorded animations. Respondents were asked to assess eachvideo. 30 people responded entirely to the questionnaire.

The Kinect camera was positioned in front of the monitor,under the screen (See Fig. 5). User’s face distance from theKinect was kept around 65cm, which is the optimal distancesuggested by the FaceShift software. A mirror was providedto help users during the task.

As direction material, we selected 12 movie clips for theusers’ task. Six of them, showing the 6 Ekman’s emotions[7], are taken from the DaFEx database of dynamic facialexpressions [3, 2] (Actor 1, high intensity). The actor isa professional, his gender is female and intensity 1 can beconsidered as mild. Three other clips have been selected

Figure 5: The user study setup.

from the “Fourteen Actors Acting: a video gallery of classicscreen types” recorded by the New York Times in 2010 [34].We chose the clips featuring Cloe Moreetz, Robert Duvalland Michael Douglas. Finally, the last three clips are close-up sequences of Fritz Lang’s expressionist movies: two clipsare excerpts from “Metropolis” and clip from “M”. Eachvideo clip is lasts between 8 and 15 seconds.

The second part of the study was conducted as a publiconline questionnaire (in English) that we advertised in mail-ing lists, and social networks. In total 96 participants an-swered the questionnaire, among them, 30 completed it.The questionnaire was anonymous and consisted into 57pages. The first page was introducing the questionnaireand presenting the context of the study. The second pagewas gathering anonymous information about the respondent(age, experience with computer and video game) follow-ing were 54 pages displaying expression videos (×6) (per-formed by the professional artist), the video recordings of4 users (U5;U6;U8;U10) selected among the 16 by the ex-perimenters for their “acting” talents (×(4 × 6) pages) andrendered animations of the recorded animations (×(4 × 6)pages). Respondents were asked to assess each video. along6 eckman dimensions E1 to E6 (happy, surprised, scared,sand, angry, disgusted). Each rating was done on a 6 pointscale from 0 to 5.

We analyzed the 30 participants who completely filled thequestionnaire. Fig. 6 shows the ratings obtained by thevideo recordings of the users when recording their perfor-mances. As for the rendered videos whose ratings are shownin Fig 7. The results displayed for the recorded (Fig.6) andthe rendered (Fig.7) videos share a common tendency. Thisresult, does not tell us that the recorded animations aregood enough to be used as-is for Sign Language material. Ithowever shows that novice users can use the device to pro-duce animations that are evaluated in a similar way thanthe recorded videos of the users. Needless to say, futureSign Language centered studies will follow.

5. CONCLUSION AND FUTURE WORKTo sum up, we have presented an animation system thathas the potential, on the one hand, to enable novice users toauthor complex animations of humanoid character and onthe other hand, to increase the productivity of experienced

Figure 6: Rating obtained for the recorded videos (subjects) along each expression by each dimension

Figure 7: Rating obtained for the rendered videos (avatar), along each expression by each dimension

users. This system is built upon both the Leap Motion anda kinect-like device. It provides a natural animation au-thoring interface that supports two different edit schemes:performance capture and pose-to-pose animation. In twouser studies, we could show, on the one hand that this sys-tem lets a novice user perform non-trivial positioning andposing task as fast a an experienced user would do using atraditional Keyboard and Mouse interface. When handledby an experienced user on the same tasks, the system dou-bles his velocity when compared to the classical Keyboardand Mouse input. On the other hand, for facial performancecapture, we could show that novice users can use Kinect-likedevices to produce animations that are evaluated in a similarway than their recorded videos. Since all the devices we usedwill be commercially available at the time the paper mightbe published and since we used open source software to buildthis system, we are publishing on-line8 all the sources thatare necessary to build and reproduce the descried experi-ment.

This work is an intermediate step in a global project aim-ing at developing a crowd-sourced sign-language animationeditor for the many Deaf Communities. We believe that theresults presented in the paper are encouraging. In futurework, we are planning to evaluate how the two edit schemesdescribed in Section 3 might benefit from each other, bothwhen used by professional animators, and novice (regardingcomputer animation) Deaf users.

6. ACKNOWLEDGMENTS8http://slsi.dfki.de/software-and-resources/ (28 July 2013)

This research has been carried out within the framework ofthe Excellence Cluster Multimodal Computing and Interac-tion (MMCI) at Saarland University, funded by the GermanResearch Foundation (DFG).

7. REFERENCES[1] N. Adamo-Villani, V. Popescu, and J. Lestina. A

non-expert-user interface for posing signing avatars.Disability and Rehabilitation: Assistive Technology,8(3):238–248, May 2013.

[2] A. Battocchi, F. Pianesi, and D. Goren-Bar. DaFEx:database of facial expressions. In Proceedings of theFirst international conference on IntelligentTechnologies for Interactive Entertainment,INTETAIN’05, page 303aAS306, Berlin, Heidelberg,2005. Springer-Verlag.

[3] A. Battocchi, F. Pianesi, and D. Goren-Bar. A firstevaluation study of a database of kinetic facialexpressions (DaFEx). In Proceedings of the 7thinternational conference on Multimodal interfaces,ICMI ’05, page 214aAS221, New York, NY, USA,2005. ACM.

[4] G. Breton. Dynamixyz | expressive machines, 2011.

[5] J. Chai and J. K. jessica Hodgins. Performanceanimation from low-dimensional control signals. InProc. of ACM Transactions on Graphics (SIGGRAPH2005), 2005.

[6] M. Delorme, M. Filhol, and A. Braffort. Animationgeneration process for sign language synthesis. pages386–390. IEEE, Feb. 2009.

[7] P. Ekman. Emotion in the human face: guide-lines for

research and an integration of findings. NumberPGPS-11 in Pergamon general psychology series.Pergamon Press, New York, 1972.

[8] R. Elliott, J. R. W. Glauert, J. R. Kennaway,I. Marshall, and E. Safar. Linguistic modelling andlanguage-processing technologies for avatar-based signlanguage presentation. Univers. Access Inf. Soc.,6:375–391, January 2008.

[9] A. Heloir and M. Kipp. EMBR - a realtime animationengine for interactive embodied agents. In Proceedingsof the 9th International Conference on IntelligentVirtual Agents (IVA-09), 2009.

[10] K.-S. Huang, C.-F. Chang, Y.-Y. Hsu, and S.-N. Yang.Key probe: a technique for animation keyframeextraction. The Visual Computer, 21(8-10):532–541,Sept. 2005.

[11] M. Jemni and O. Elghoul. A system to make signsusing collaborative approach. In K. Miesenberger,J. Klaus, W. Zagler, and A. Karshmer, editors,Computers Helping People with Special Needs, number5105 in Lecture Notes in Computer Science, pages670–677. Springer Berlin Heidelberg, Jan. 2008.

[12] J. R. Kennaway, J. R. W. Glauert, and I. Zwitserlood.Providing signed content on the internet bysynthesized animation. ACM Transactions onComputer-Human Interaction (TOCHI), 14(3):15–29,2007.

[13] M. Kipp, A. Heloir, and Q. Nguyen. Sign languageavatars: animation and comprehensibility. InProceedings of the 10th international conference onIntelligent virtual agents, IVA’11, page 113aAS126,Berlin, Heidelberg, 2011. Springer-Verlag.

[14] M. Kipp and Q. Nguyen. Multitouch puppetry:creating coordinated 3D motion for an articulatedarm. page 147. ACM Press, 2010.

[15] T. Lebourque and S. Gibet. High level specificationand control of communication gestures: theGESSYCA system. In Computer Animation, 1999.Proceedings, page 24aAS35, 1999.

[16] J. P. Lewis and K.-i. Anjyo. Direct manipulationblendshapes. IEEE Computer Graphics andApplications, 30(4):42–50, July 2010.

[17] J. Lin, T. Igarashi, J. Mitani, M. Liao, and Y. He. Asketching interface for sitting pose design in the virtualenvironment. IEEE Transactions on Visualization andComputer Graphics, 18(11):1979–1991, Nov. 2012.

[18] K.-Y. Liu, W.-C. Ma, C.-F. Chang, C.-C. Wang, andP. Debevec. A framework for locally retargeting andrendering facial performance. Computer Animationand Virtual Worlds, 22(2-3):159–167, Apr. 2011.

[19] N. Lockwood and K. Singh. FingerWalking: motionediting with contact-based hand performance. InProceedings of the 11th ACM SIGGRAPH /Eurographics conference on Computer Animation,EUROSCA’12, page 43aAS52, Aire-la-Ville,Switzerland, Switzerland, 2012. EurographicsAssociation.

[20] V. Lombardo, F. Nunnari, and R. Damiano. A virtualinterpreter for the italian sign language. InProceedings of the 10th international conference onIntelligent virtual agents, IVA’10, pages 201–207,Berlin, Heidelberg, 2010. Springer-Verlag.

[21] O. Losson and J.-M. Vannobel. Sign language formaldescription and synthesis. In Proc. 2nd Euro. Conf.Disability, Virtual Reality and Assoc. Tech., pages129–137, 1998.

[22] P. Lu and M. Huenerfauth. Collecting an americansign language corpus through the participation ofnative signers. In D. e. a. Hutchison, editor, UniversalAccess in Human-Computer Interaction. Applicationsand Services, volume 6768, pages 81–90. SpringerBerlin Heidelberg, Berlin, Heidelberg, 2011.

[23] A. Menache. Understanding Motion Capture forComputer Animation and Video Games. MorganKaufmann Publishers Inc., San Francisco, CA, USA,1999.

[24] M. Neff, I. Albrecht, and H.-P. Seidel. Layeredperformance animation with correlation maps.Computer Graphics Forum, 26(3):675–684, Sept. 2007.

[25] F. Pighin and J. P. Lewis. Facial motion retargeting.In ACM SIGGRAPH 2006 Courses, SIGGRAPH ’06,New York, NY, USA, 2006. ACM.

[26] B. Robertson. Mike the talking head. ComputerGraphics World, 11(7), 1988.

[27] T. Roosendaal. blender.org - home, 2013.

[28] A. Sanna, F. Lamberti, G. Paravati, and F. D. Rocha.A kinect-based interface to animate virtual characters.Journal on Multimodal User Interfaces, Oct. 2012.

[29] A. Sears and B. Shneiderman. High precisiontouchscreens: design strategies and comparisons witha mouse. International Journal of Man-MachineStudies, 34(4):593–613, Apr. 1991.

[30] J. Seo, G. Irving, J. P. Lewis, and J. Noh.Compression and direct manipulation of complexblendshape models. ACM Transactions on Graphics,30(6):1, Dec. 2011.

[31] H. J. Shin, J. Lee, S. Y. Shin, and M. Gleicher.Computer puppetry: An importance-based approach.ACM Transactions on Graphics, 20(2):67–94, Apr.2001.

[32] W. C. Stokoe. Sign language structure: An outline ofthe visual communication system of the Americandeaf. Studies in linguistics, Occasional papers, 8, 1960.

[33] M. Thorne, D. Burke, and M. van de Panne. Motiondoodles. ACM Transactions on Graphics, 23(3):424,Aug. 2004.

[34] N. Times. The hollywood issue - 14 actors acting -james franco, natalie portman, matt damon and more- NYTimes.com, 2010.

[35] T. Weise, S. Bouaziz, H. Li, and M. Pauly. Realtimeperformance-based facial animation. In ACMSIGGRAPH 2011 papers, SIGGRAPH ’11, page77:1aAS77:10, New York, NY, USA, 2011. ACM.

[36] R. Wolfe, J. McDonald, M. J. Davidson, and C. Frank.Using an animation-based technology to supportreading curricula for deaf elementary schoolchildren.In The 22nd Annual International Technology &Persons with Disabilities Conference, 2007.