MotionDraw: a Tool for EnhancingArt and Performance Using Kinect

Danilo Gasques RodriguesUniversidade Federal de Sao Carlosdanilogr@comp.ufscar.br

Marcel de Sena Dall’AgnolUniversidade de Sao Paulomarcel.agnol@usp.br

Emily GrenaderUniversity of California San Diegoegrenade@mail.ucsd.edu

Troels E. HansenIT University of Copenhagentroe@itu.dk

Fernando da Silva NosPUC do Rio Grande do Sulfernando.nos@acad.pucrs.br

Nadir WeibelUniversity of California San Diegoweibel@ucsd.edu

Figure 1: Abbie Wang dances with MotionDraw.

Copyright is held by the author/owner(s).CHI 2013 Extended Abstracts, April 27 – May 2, 2013, Paris,France.ACM 978-1-4503-1952-2/13/04.

AbstractContemporary staged performances frequently utilizeadvanced lighting and projection techniques. The designand creation of these stage effects are rarely accessible toactual performers and must be designed by professionallighting designers or highly-paid programmers. WithMotionDraw we want to create an affordable system thatis easily controlled and manipulated by performers. Withintuitive gestures, non-specialized users can control theMotionDraw visual library and interact with the capturedvisual record of their own movements. Possible uses of oursystem grew out of research with dancers and performers,and the current technical implementation sets a frameworkfor including additional visual libraries and capabilities.

Author KeywordsProjection; drawing; real-time; Kinect; interactivity;design; experimentation; performance.

ACM Classification KeywordsH.5.m [Information Interfaces and Presentation]:Miscellaneous; J.5 [Arts and Humanities]: Performing arts.

IntroductionComputer graphics, projection techniques, animations,and professional lighting enable effective stage effects inlarge-scale events that support the combination of live

Work-in-Progress: Gesture CHI 2013: Changing Perspectives, Paris, France

1197

performance, such as dance, and digital arts1,2,3.However, creating those exciting environments comes withthe high cost of specialized hardware and often withdedicated software developed by professionalprogrammers. MotionDraw, a Kinect-based tool thataugments performance with projected visualizationscreated from tracked live movements of users, enablesartists, performers, dancers and even the audience toeasily design, create and control hybrid digitalperformances. In this paper we present the motivationand research that led to the development of MotionDraw,including a historical perspective on visual and digital arts.We describe the technical implementation of our systemand the results of its deployment with dancers.

Figure 2: A concept drawing forMotionDraw by Emily Grenader.

Figure 3: From the Performance ”Apparition” by KlausObermaier and Ars Electronica Futurelab

Motivation, Inspiration, and ResearchInspired by the way photography began as a scientifictool, and later became an art form, we wanted to explorehow the Kinect tool could be used to enable art and

1“Apparition” by Klaus Obermaier and Ars Electronica Futurelab,http://www.exile.at/apparition, Fig. 32“Living room” by Recoil Performance, http://goo.gl/inDto.3“Mortal Engine” by Chunky Move,http://www.youtube.com/watch?v=sbjOMualLVs.

performance. In his photograph from 1884, shown inFig. 4, Etienne-Jules Marey tracks a man’s motion usingmultiple exposure photography. Even though the rationaleof creating this photograph was a scientific study ofmotion, it is evident that the result is also an aestheticallyinteresting image.

Figure 4: “Chronophotographic Image Formed by ManWalking in Black Suit with White Stripes”, Etienne-JulesMarey, 1884

Figure 5: Theman in suitphotographedin Fig. 4

To aid us in the design of our application, develop andvalidate our idea, we conducted small research interviewswith seven dancers from UCSD’s dance department. Weasked the dancers about their prior experience with thecombination of dancing, projection and computing. Allinterviewees had seen large-scale performances that usedtechnology and/or projection to enhance the performance,but none had been given the opportunity to work withcomputing or projection themselves, even though most ofthem expressed interest in doing so. Given that dancersusually have very little control over what is happening onthe stage, we saw this as further opportunity to developan application that would be easy to use for performers,therefore changing the established system. From thebeginning our ultimate goal was to enable professionalperformers, students, and at-home users to create visualinteractive effects that they can easily control withintuitive gestures.

Work-in-Progress: Gesture CHI 2013: Changing Perspectives, Paris, France

1198

The dancers we interviewed were interested in real-timecreation of visual effects—projections on a stage thatrespond to their movements—but they also showedinterest in post-processing possibilities that wouldaugment a recorded performance with digitalartifacts—such as fine tuning the data captured from theirdance by adding effects in external 3D applications. Theidea of drawing with movement, as proposed by us (bodymovements being drown as ”brush strokes” on screen)was well accepted by the interviewees.

We were surprised that most of the dancers would notmind having another person conducting theirperformance. Many of them thought that improvising incollaboration with an external conductor would be a greatexperience, and actually preferred this set-up, wanting toalso experiment with acting as the conductor themselves.As expected, our interviews showed that practicingdancers are very interested in enhancing their performancewith interactive projection and computing, but that thetechnology to do this is not easily accessible to them atthe moment. Each of the dancers we interviewed thoughtthat drawing with motion using Kinect technology andhaving a conductor that would direct part of theperformance would be an attractive solution to thisproblem. We will describe how we achieved this goal inthe remainder of this paper, after situating our workwithin previous research in this setting.

Related WorkPrevious projects use visual displays in order to augmentperformance. For example, dancer and choreographerMerce Cunningham frequently worked with digital artistsand musicians to place dancers into dynamic visual

environments. In Hand-drawn Spaces4, Cunninghamworked with the OpenEndedGroup to create displays ofmultiple abstract figures across many screens usingmotion capture. In 2002 during a performance at PurdueUniversity, Meader and colleagues explored interactivedance with the use of a motion-capture suit worn by adancer that created a projected, mirror-image visualizedcharacter. This character, mimicking the moves of thereal dancer, interacted by becoming abstracted, changingsize, and using different camera angles [1]. In this kind ofperformance, although the choreographer can be in closeconversation with the computer operators and graphicartists, dancers and choreographers can not easily makechanges to the graphic display on their own.

Projects like Sandde5 allow for three-dimensional digitaldrawing using gesture, but rely on hardware (a magneticpen) to track movement. A project by Raptis et al. [2]focuses on the recognition and classification of a set ofpredetermined dance gestures. While this work is relatedto computer vision research, the ability to recognizespecific dance moves provides an interesting feature.

Since the release of the Microsoft Kinect device, therehave been several efforts towards integrating art andmotion-tracking technology. Super Mirror [3] brings thelow-cost possibility of using the Kinect into danceinstruction, functioning as an “augmented mirror”.Alexiadis et al. [4] used Kinect technology for a similarproject that provides real-time evaluation of dancers withvisual feedback, and also supports evaluation of onedancer’s performance against another. Both of theseprojects have instructional or evaluational purposes, and

4“Hand-drawn Spaces” by Merce Cunningham and OpenEndedGroup,http://openendedgroup.com/artworks/hds.html.5“Sandde - Stereoscopic animation drawing device”,http://www.sandde.com.

Work-in-Progress: Gesture CHI 2013: Changing Perspectives, Paris, France

1199

are not aimed at performances. Crossole by Senturk et al.integrates gesture tracking and sound manipulation, whichallows the user to modify characteristics of the music [5].Kinect has also been integrated into professionalperformances as in .cyclic., a collaboration betweenUniversity of Waikato and Stellaris dance in New Zealand,where choreographers and computer scientists created aninteractive performance [6]. This project depends highlyon each individual collaborator’s ability to complete atask, and remains inaccessible to everyday users.

Figure 6: Two-Kinect set-up

The MotionDraw SystemMotionDraw consists of a set-up with one or two Kinects,that record the position of joints of a person’s body overtime and draws the resulting trail on a 3D canvas. OneKinect points at the dancers and tracks their movements.A GUI-based control panel, enables a separate user tochange the parameters of the projected drawing inreal-time, such as (a) change the color and width of thetrails, (b) move the position of the camera-view, and (c)enable or disable the visualization of specific jointstracked. The program also saves the Kinect stream to afile for further post-production.

With a two-Kinect set-up (Fig. 6) the second Kinectcaptures a ”conductor” (from here on, referred to as theKinecTor) who can interact directly with the MotionDrawsystem and change the visualization parameters throughgestures. This integration is covered in more details below.

MotionDraw is designed to be simple and extensible. Inour system, users can easily control the interface indifferent ways, communicate with external software, andexport the recorded data for further processing. In theremainder of this section we explain the differentcomponents behind the system, while Fig. 7 illustratesMotionDraw’s main software architecture.

Graphical User InterfaceThe user interface is built using OpenFrameworks,specifically, using a plugin named ofxUI 6. It is composedof one window, on which it is possible to switch from aViewer perspective to a Control Box perspective, andvice-versa. A 3D drawing is presented in the Viewerwindow and users have options to select different Kinectdevices in the Control Box window (Fig. 9). Here, usersare able to change options such as color, width, tail andvisibility, for all skeleton joints. The 3D drawing on theviewer window is achieved by processing data receivedfrom a Kinect device using OpenGL.

Figure 7: MotionDraw architecture. The Kinects communicatedirectly with MotionDraw (the Kinect tracking the dancer) orover an IPC interface (one or more KinecTors, on the right).The OpenGL interface (on the left) is responsible for thegraphical representation that is projected on the stage.MotionDraw also supports directly exporting the tracking ofthe joints to external tools such as Blender and Maya forpost-processing, through dedicated Python scripts (bottom).

6Filip - “ofxUI” a new GUI addon for openFrameworks projects,http://goo.gl/NLxUt

Work-in-Progress: Gesture CHI 2013: Changing Perspectives, Paris, France

1200

KinecTorThe modular architecture enables external plugins tocommunicate with the MotionDraw application andcontrol the main visualization. One of these plugins is theKinecTor. As in an orchestra, the KinecTor works as aconductor, enabling users (i.e., dancers) to participate inthe performance by changing the way the dancers inputsare processed. By using a dedicated Kinect to implementthe KinecTor, we enable users to interact with the systemdirectly with gestures (as a conductor) without having touse a standard computer interface based on mouse andkeyboard. This approach further supports the hands-freeparadigm expanding it to the conductor as well. Figure 8illustrates the use of the KinecTor and the gesturesemployed by the conductor.

While the KinecTor represents one way to tunevisualization parameters, the flexible architecture ofMotionDraw makes it possible to use other input devicesor other interaction metaphors as an interface to theprojected visualization. We envision a range ofpossibilities in this setting and discuss this further in theFuture Work section.

Figure 8: KinecTor gestures.

Inter-Process CommunicationIn order to support two or more Kinect devices interactingwith MotionDraw through the KinecTors, we developed ageneral Inter-Process Communication (IPC) interface thatenables any outside applications (Kinect or non-Kinectbased) to control the way MotionDraw processes theuser’s inputs. We based the IPC implementation onsockets and dedicated messages passed across machinesinterconnected through a network.

Post-ProcessingMotionDraw also allows users to create self-containedrecordings of performances and work offline, using a log of

tracked movements as recorded by the Kinect. In this way,MotionDraw makes it possible to use different softwarepackages for post-processing purposes, such as importingthe recorded data into tools, like Blender or Maya.

Results, Initial Evaluation and Future WorkWe employed MotionDraw in a live performance with twodancers, Tony Ho and Abbie Wang, who volunteered toperform for a few people accompanied by a solid set-upcontaining music and a projector for visual effects. Oursystem was used to track the movements of both dancers.An opera singer also accompanied the performance, whileacting at the same time as the conductor, using ourKinecTor interface.

Figure 9: MotionDraw Control interface

Although the event was informal, it was possible to gatherinitial feedback from both the audience and the dancers.We found that MotionDraw did indeed provide theperformance with an additional level of interaction andvisual effects, which both the audience and the dancersfound to be pleasing. However, since the system is in an

Work-in-Progress: Gesture CHI 2013: Changing Perspectives, Paris, France

1201

early stage of development, the range of possible effectsand interactions that we could implement was limited.Nevertheless, the flexibility and extensibility ofMotionDraw enable the creation and easy integration ofnew effects both for real-time and post-processing.Thesecan be added to the existing library of visualizations andused by novice KinecTors.

Figure 10: Point cloudrepresentation generated withMotionDraw.

We are further developing the graphical and interactivefunctionality in MotionDraw. In particular we would liketo support following functionality:

• Interactivity among dancers (e.g. touch of hands)as a trigger for changes in the display.

• Interacting with the actual projection - touching theprojected drawing and influencing the projection.

• Relate the position of the projection to the body’sposition in order to create a shadow or completelycover the performer with the drawing.

• Improve real-time graphics with filters or pointcloud effects (see Fig. 10 shows an example).

We are also considering alternative scenarios where theaudience could be directly involved as the KinecTor or twodancers as each other’s KinecTors. Figure 11 illustratesthose scenarios. Additionally, MotionDraw would also besuitable for use via the Web to bring people from differentplaces together in an online dance collaboration.

AcknowledgementsWe would like to thank all of the dancers we interviewed,especially Tony Ho and Abbie Wang, as well as MicrosoftCorp. that donated the Kinects, the ExperimentalDrawing Studio, the Department of Computer Scienceand Engineering and the Moxie Center at UCSD, and theBrazilian Science Without Borders program, for providingus with the necessary support for this project.

Figure 11: Future set-up

References[1] M. W. Scott, et al. ”Mixing dance realities:

collaborative development of live-motion capture in aperforming arts environment.” Proc. CIE 2004:12-12.

[2] M. Raptis, D. Kirovski, and H. Hoppe. ”Real-timeclassification of dance gestures from skeletonanimation.” Proc. SCA’11.

[3] M. Zoe, et al. ”Super Mirror: a kinect interface forballet dancers.” Proc. CHI’12.

[4] D. S. Alexiadis, et al. ”Evaluating a dancer’sperformance using kinect-based skeleton tracking.”Proc. Multimedia’11.

[5] S. Senturk, et al. ”Crossole: A Gestural Interface forComposition, Improvisation and Performance usingKinect.”

[6] D. Jung, et al. ”.cyclic.: interactive performancecombining dance, graphics, music andkinect-technology.” Proc. NZ Chapter SIGCHI, 2012.

Work-in-Progress: Gesture CHI 2013: Changing Perspectives, Paris, France

1202