F L O O R M A P P E R , A N I N T E R A C T I V E M A P I N S TA L L AT I O N U S I N G T H EM I C R O S O F T K I N E C T

john wika haakseth

TDT69 - Artistic Software

Faculty of Information Technology, Mathematics and Electrical EngineeringDepartment of Computer and Information Science

November 2012

John Wika Haakseth: FloorMapper, an Interactive map installation usingthe Microsoft Kinect, TDT69 - Artistic Software, c© November 2012

Course coordinator: Letizia Jaccheri

A B S T R A C T

FloorMapper is an interactive installation developed by a student aspart of a specialization project of Geomatics in the fall of 2012 atNTNU. It is a mapping application that projects a large map ontothe floor and allows people walking by to interact with it using theirbodies and gestures.

The hardware part of FloorMapper consists of an Xbox Kinect thattracks the audience’s joints as they cross the map, a standard projectorthat displays the map and a computer that runs the application. Thesoftware was developed using the Processing IDE and language withthe SimpleOpenNI and Unfolding Maps libraries.

FloorMapper can be described as a piece of computer art and as asoftware engineering product, with defined stakeholders. The inter-action categorization of FloorMapper are static, triggered by humansand pre-defined content, that may be dynamically manipulated bythe audience.

iii

C O N T E N T S

i preface 1

1 introduction 3

1.1 Related work . . . . . . . . . . . . . . . . . . . . . . . . . 3

ii materials and methods 5

2 development 7

2.1 Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.2 FloorMapper in a Computer Art Context . . . . . . . . 7

3 hardware components 9

3.1 The Microsoft Kinect . . . . . . . . . . . . . . . . . . . . 9

3.1.1 Introduction . . . . . . . . . . . . . . . . . . . . . 9

3.1.2 Technological specifications, motion sensing prin-ciples and accuracy . . . . . . . . . . . . . . . . . 10

4 software components 11

4.1 Processing . . . . . . . . . . . . . . . . . . . . . . . . . . 11

4.1.1 Programming in Processing . . . . . . . . . . . . 12

4.1.2 Libraries in Processing . . . . . . . . . . . . . . . 13

iii result and conclusion 15

5 floormapper 17

5.1 Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

5.2 Features . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

5.2.1 Hotbox based . . . . . . . . . . . . . . . . . . . . 18

5.2.2 Skeleton tracking based . . . . . . . . . . . . . . 19

6 conclusion 21

bibliography 23

v

L I S T O F F I G U R E S

Figure 1 The Processing IDE . . . . . . . . . . . . . . . . 12

Figure 2 GraduatedColors . . . . . . . . . . . . . . . . . 13

Figure 3 Hardware setup of FloorMapper. . . . . . . . . 17

Figure 4 Interactive floor map. . . . . . . . . . . . . . . . 18

Figure 5 Illustration of the concept of hot boxes. . . . . 19

Figure 6 Area under foot displays aerial photo. . . . . . 20

L I S T O F TA B L E S

Table 1 Hardware components of FloorMapper . . . . 9

Table 2 Software components of FloorMapper . . . . . 11

vi

A C R O N Y M S A N D T E R M I N O L O G Y

Kinect/The kinect Refers to the (depth) sensor Microsoft Kinect.Currently, there are two models: One designed for the Xbox360 gaming console, the Xbox Kinect, and one designedspecifically for Windows computers, the Xbox for Windows.

Processing Open source Java based framework for creatingcross-platform applications, focusing on images, animationsand interactions.

Unfolding/Unfolding Maps Processing library that allows displayingand interacting with maps.

NI Natural Interaction

OpenNI Non-profit organization focused on improving andcertificating frameworks and APIs for development for naturalinteraction devices.

SimpleOpenNI Processing library for accessing the data from theKinect.

SDK Software Development Kit: Set of tools to aid development ofapplications of a specific purpose.

API Application Programming Interface: Specification intended tobe used as an interface for communication between softwarecomponents.

NTNU Norwegian University of Science and Technology

IVT Faculty of Engineering Science and Technology (Fakultet foringeniørvitenskap og teknologi)

BAT Department of Civil and Transport Engineering (Institutt forbygg, anlegg og transport)

vii

Part I

P R E FA C E

This paper is written as part of theory module, TDT69,at the Department of Computer and Information Scienceand the Faculty of Information Technology, Mathematicsand Electrical Engineering at the Norwegian University ofScience and Technology in the fall of 2012.

November 2012

John Wika Haakseth

1I N T R O D U C T I O N

A girl, attending a science fair, notices a giant map projected ontothe floor of the convention center. As she crosses it, the area belowher turns into a "looking glass" into the real world as it displays anorthophoto in the area just underneath her feet. She is told to raiseher arm over her head, and as she does, icons appear in a menu onthe map. She points at an arrow symbol and as she gestures withher arms, the map pans to her desired location. She then chooses thecrosshair symbol and points to her school. A marker is placed at thatlocation.

The previous paragraph describes one of the possible scenariosof the FloorMapper installation. It was designed and developed bythe author of this paper as part of the course TBA4560 – Geomatics,Specialization Project during the fall semester of 2012 at the Norwe-gian University of Science and Technology. Supervisors for the projectwere Professor Terje Midtbø and Alexander Nossum of the Depart-ment of Civil and Transport Engineering (BAT).

The Microsoft Kinect is a low-cost technology for three dimen-sional, controller-free gaming technology. It’s great availability hasmade it popular within research environments and artst- and maker-communities since its release in 2010.

This project focused on implementing prototypes of interactive map-ping installations (FloorMapper), as well as discussing various con-cepts of interaction. This paper also draws some lines into the com-puter art domain as proposed by Oates [14].

1.1 related work

Many projects working with the Kinect and similar sensors exist al-ready. ??? remove if no time.

3

Part II

M AT E R I A L S A N D M E T H O D S

2D E V E L O P M E N T

2.1 context

The installation was developed by the author of this paper as partof the course TBA4560 – Specialization Project in Geomatics at theFaculty of Engineering Science and Technology (IVT) at the Norwe-gian University of Science and Technology (NTNU) in the fall of 2012.Supervisors for the project were Prof. Terje Midtbø and AlexanderNossum of the Department of Civil and Transport Engineering (BAT)at NTNU.

BAT is a part of a recruitment project initiated by Geoforum thatis aimed towards high school students to get more people to studyGeomatics. In this context, the institute has installations relating tothe subject at stands at various education and science fairs, such asResearcher’s Night [13] and a Norwegian higher education fair (Ut-danningsmessa) [11].

Recognizing the kinect as a possibility to blend Geomatics-relatedtopics such as laser scanning and map interaction, a project topic wasformulated with the possibility of contributing to these recruitmentefforts.

"Microsoft Kinect for use in interactive map installations" was theworking title. The result of this is FloorMapper, an interactive mapinstallation designed for use at fairs and other recruitment events.

2.2 floormapper in a computer art context

Oates [14] suggests looking at computer art as information systems,stating that it is not desirable to draw boundries around the conceptof information systems as the different contexts are not fully knownand continue to evolve.

Trifonova, Jaccheri and Bergaust [16] takes this cue to provide aroadmap for artists and engineers to communicate. They talk aboutinteractive installations as software engineering products. This in-cludes the final product, which consists of both the hardware andsoftware components of the installation, as well as documentation.These are presented in chapters 3 and 4.

The different stakeholders of a project are identifined. The artist, whocomes up with the idea of, as well as the global view of, the project.Software/hardware engineer(s) converts the artist’s vision into require-ments that later will be implemented into the final product. The audi-

7

8 development

ence, who participate in the final stage, either by interacting with theartwork, or as passive onlookers.

The artist, in the case of FloorMapper, can be defined as both thestudent implementing it and the supervisor of the project, as the su-pervisor came up with the concept, but the student developed it fur-ther with help from the supervisor. The same can be said of the en-gineer role, but with the student having most of the responsible. Theaudience is typically high school students or other interested guestsattending the events where the installation is featured.

[16] also talks about three perspectives of interaction to take intoconsideration:

what content is shown to the user?

what triggers interaction?

how is the artwork influenced by the surroundings?

Using these properties we can describe the interactive artwork bythe following three perspectives of interaction. These are interactionrules, triggering parameters and content origin, and are described in5.2.2, which presents the final product.

The tools and development environment are presented in the fol-lowing chapters.

3H A R D WA R E C O M P O N E N T S

To make sure the installation is easy to use, the needed hardware isheld to a minimum. The prototypes in part 3 consist of:

• Computer

• Kinect

• Projector

The installation should be able to run on a computer running anyof the three main operating systems (Windows, MacOS, Linux) asboth Processing and SimpleOpenNI work on all three, more on thisin chapter 4. The kinect have to be the version designed for the Xbox360 and not the newer Kinect for Windows.

The specific hardware used during the development are as follows:

Table 1: Hardware components of FloorMapper

Computer Kinect Projector

MacBook Pro runningMacOS X Lion (10.7.5)

Kinect for Xbox 360 Dell 3200MP

3.1 the microsoft kinect

3.1.1 Introduction

Originally known as Project Natal [15], the Microsoft Kinect is a 3Dcapable motion sensor, designed to allow users to use their bodiesas controllers when playing specifically designed games on the Xbox360 console.

Released in November 2010, it quickly gained popularity amongusers and currently holds the Guinness’World Record in the category"fastest selling gaming peripheral" with an average of 133.333 unitssold per day over the first 60 days [3].

Independent development interest in this "new"1, affordable tech-nology was high and at the kinect’s release Adafruit [1], an open

1 "New" because similar technology has existed for years, but not aimed at consumers

9

10 hardware components

source DIY electronics company, announced a price of $3.000 to who-ever released the first open source driver for the kinect. The first wasreleased within a week.

Initially, Microsoft stated that they do not condone third-party ven-dors tampering with the kinect in this way, but later changed theirstatement, saying they intentionally left the USB connection open toallow to read the inputs from the sensor [6].

Microsoft provides it’s own kinect SDK [2], which works with bothof the kinect versions. However, it is proprietary and works only withWindows. There are several open-source libraries available

For this project, SimpleOpenNI [7] was used. SimpleOpenNI waschosen as it is cross-platform and designed for Processing. It is furtherpresented in Chapter 4.

3.1.2 Technological specifications, motion sensing principles and accuracy

The kinect measures depth using the triangulation principle [8, 9].It consists of an infrared laser emitter, an infrared camera and anRGB camera. The emitter projects a beam that is split into a constantpattern of light onto the scene. The pattern is then captured by theIR camera and compared to that of a reference pattern stored in thesensor.

The kinect is intended to be used at a distance of between 1.2 −3.5m, but captures data beyond that and as close as 0.7m. Khoshel-ham and Oude Elberink [9] presented an accuracy analysis of thekinect and compared the output data to that of a high-end laserscanner point-cloud. It concluded that the random error increasesand depth resolution decreases quadratically with increasing distancefrom the sensor. The error reaches a maximum of 4cm at 5m whilstthe resolution is 7cm at this distance.

4S O F T WA R E C O M P O N E N T S

Two prototypes were created over the course of the project. The differ-ence between these were mainly the interaction concepts by which theuser controlled the map. In this report "the application" or "FloorMap-per" can refer to either.

Table 2: Software components of FloorMapper

Name Description

FloorMapper Applications written in Processing (about 300 linesof code).

SimpleOpenNI Processing library for accessing kinect data.

UnfoldingMaps Processing library for displaying and interactingwith maps.

4.1 processing

Processing [4] is an open source, cross-platform (Windows, MacOS,Linux and more) Java-based language and IDE 1 for developing cre-ative applications that focus on visualizations, animations and inter-actions.

The strength of Processing is that it is fairly simple to get started,but it still offers full Java functionality and the possibility to add li-braries to extend its functionality.

1 Processing also comes as a library to be imported into any Java-application

11

12 software components

Figure 1: The Processing IDE

4.1.1 Programming in Processing

At the base of any Processing application lies two methods, setup()and draw(), and understanding the use of these is the key to masteringProcessing.

setup() is called one time when the application starts. draw() iscalled after setup(), and then once every time the application windowupdates (the default frame rate of a Processing application is 60 fps,but this can be set manually by calling framerate(int framerate)).

The code in figure 1 makes a simple application that displays awindow of size 200x200 pixels where the background color gradu-ally changes from black to red depending on the cursor’s locationhorizontally over the window as seen in figure 2.

4.1 processing 13

Figure 2: GraduatedColors, the resulting application from figure 1

4.1.2 Libraries in Processing

Processing, as Java, enables the user to import libraries in order to adddesired functionality. In addition to OpenGL, which comes bundledwith Processing, two libraries were used for FloorMapper.

Unfoldning Maps [12] is a library that adds mapping functionalitythat allows for common map interactions, such as panning, zoomingand rotating maps. The maps themselves are tile-based, and can bedesigned by the developer, using a service like TileMill [10].

SimpleOpenNI is an open-source library for Processing, that allowsProcessing to read the data from the kinect.

Part III

R E S U LT A N D C O N C L U S I O N

5F L O O R M A P P E R

The result of this project is FloorMapper, an interactive map installa-tion to be used at education fairs and other arenas where BAT wantsto market Geomatics.

At the time of this writing, two prototypes have been made, usingdifferent concepts for interaction. The second one, based on skeletontracking, described in Section 5.2.2, is the one that is closest to the"final product". Common for both of them is the setup and basic ideaprojecting a map on to the ground where people can pass by, havingthe map react to user interactions.

5.1 setup

As described in Chapter 3, the setup is made up of three components:The kinect, a projector and a computer. The projector is mounted highon a tripod, and angled down towards the ground in front of it, sothat the map appears about 2 meters in front of the projector.

Figure 3: Hardware setup of FloorMapper.

17

18 floormapper

The kinect is then placed on the ground , or on a low tripod under-neath the projector tripod, facing towards the map. The location ofthe kinect needs to be fine-tuned after the application has launchedfor some of the features to work properly.

The computer is kept on a table behind the tripod, allowing curiousstudents to have a closer look at the code or look at the kinect.

5.2 features

5.2.1 Hotbox based

Developed in time for the annual Researcher’s Night [SOURCE] atNTNU, this application displays a map on the ground where peopleare expected to walk by, and reacts as they enter one (or more) ofeight "boxes" defined in the space above the map.

Figure 4: Interactive floor map.c©Alexander Nossum

The interaction method used is described by Borenstein [5]. Thekinect is able to record a live point cloud. A hot box is then a boxdefined in space (in this case, relative to the kinect), that counts how

5.2 features 19

many points are located within itself. The application is then able tostart events as enough points are found within the box.

Figure 5: Illustration of the concept of hot boxes

For the FloorMapper, 8 boxes were defined in the area above themap (distributed like the numeric keypad (numpad) on a standardkeyboard, excluding the number 5). Then, as people would pass overthe map, the map would react and grab their attention.

The user is then able to stand in the middle of the map (at "number5") without anything happening, and take a step toward any of theother "buttons" to interact with the map. This application used themetaphor of the arrow keys on the numpad to allow the user to panthe map up (8), down (2), left (4) and right (6). At the boxes corre-sponding to 7 and 9 on the numpad, the map would rotate left andright, while 1 would reset it (as it was easy to get lost while panning).

5.2.2 Skeleton tracking based

This version takes advantage of SimpleOpenNI’s skeleton tracking ca-pabilities for more natural interactions. When a user is calibrated, thekinect is able to track all the main joints, which allows for more natu-ral interactions. The calibration requires the user to stand awkwardlyin front of the kinect with his hands in the air, but this can be by-passed by bundling a pre-made calibration file with the application,so this has been done here.

When a user enters the space and is automatically calibrated, tonotify him that "something is happening", an area underneath hisfoot shows the corresponding area as an aerial photo, using a sort ofloupe metaphor as shown in the screenshot in figure 6.

The user can then raise his arm over his head to call up a controlpanel. This displays four different tools the user can then point at toselect:

20 floormapper

Figure 6: Area under foot displays aerial photo.

pan : When the pan tool is active, the user either swipe with his armsor take a step towards a direction to allow the map to move.

zoom : This tool allows the user to zoom in our out in the map us-ing "pinch" motions, known to users of touch surfaces, with hisarms.

rotate : The rotate tool allows the user to rotate the map by holdinghis arms up and moving them opposite directions over the map.

place marker : When this tool is selected, a place marker is put un-der the user’s shoe, and the coordinates of this location is storedin a text file. The idea is that users can pin their hometown or aplace dear to them on the map, and these can be presented lateron a web page.

The control panel has not been implemented visually at the timeof this writing, but the user now instead points to one of the fourcorners to select one of the tools.

5.2.2.1 Interaction

Trifonova et al. [16] lists three categories of interactions in interactiveinstallation art, as described in section 2.2. For FloorMapper they are:

interaction rules : The interactions of FloorMapper are all pre-designed, so the interaction rules are static.

triggering parameters : The application triggers to human ac-tions (a person crosses the map space, is calibrated by the kinectand the area under the person appears as an aerial photo).

content origin : FloorMapper has pre-defined content, that maybe dynamically manipulated by the audience as they add mark-ers to the map.

6C O N C L U S I O N

This paper have presented the project FloorMapper, an interactivemapping application that allows the user to interact with a map pro-jected onto the floor using gestures. It puts the project into a computerart context, categorizing its interaction methods as described by Tri-fonova et al [16].

The Microsoft Kinect is presented as an exiting tool that opens upnew possibilities for creators in ways previously impossible.

21

B I B L I O G R A P H Y

[1] Adafruit industries, unique & fun diy electronics and kits. web.URL http://www.adafruit.com.

[2] Download the kinect sdk & developer toolkit | kinectfor windows. URL http://www.microsoft.com/en-us/

kinectforwindows/develop/developer-downloads.aspx.

[3] Fastest-selling gaming peripheral, guinness world records,January 2011. URL http://www.guinnessworldrecords.com/

records-9000/fastest-selling-gaming-peripheral/.

[4] Processing.org, September 2012. URL http://www.processing.

org.

[5] Greg Borenstein. Making Things See: 3D vision with Kinect, Pro-cessing, Arduino, and MakerBot. Make: Books. O’Reilly Media, In-corporated, 1 edition, 2012. ISBN 9781449307073. URL http:

//makingthingssee.com.

[6] Josh Catone. Microsoft now "excited" by kinect hacks.Web, November 2010. URL http://mashable.com/2010/11/20/

microsoft-kinect-hacks/.

[7] Google Code. Simple-openni. Google Code. URL https://code.

google.com/p/simple-openni/.

[8] Barak Freedman, Alexander Shpunt, Meir Machline, and YoelArieli. Depth mapping using projected patterns, 2012.

[9] Kourosh Khoshelham and Sander Oude Elberink. Accuracy andresolution of kinect depth data for indoor mapping applications.Sensors, 12:1437–1454, 2012. doi: 10.3390/s120201437.

[10] MapBox. Tilemill | fast and beautiful maps. Web. URL http:

//mapbox.com/tilemill/.

[11] Marianne Meinich. Geomatikkbransjen på utdanningsmessen.Web, January 2012. URL http://www.geoforum.no/Members/

mmeinich/nyheter-2012/x-y-og-z-over-og-under-bakken/

?searchterm=lillestr\T1\om.

[12] Till Nagel. Unfolding maps: Unfolding is a library to create inter-active maps and geovisualizations in processing and java. URLhttp://unfoldingmaps.org.

23

24 Bibliography

[13] Alexander Nossum. Tråkkekart og researcher’s night en suk-sess. Blog, October 2012. URL http://alexanno.net/2012/10/

trakkekart-og-researchers-night-en-suksess/.

[14] Briony J. Oates. New frontiers for information systems research:computer art as an information system. European Journal of Infor-mation Systems, 15:617–626, October 2006.

[15] Daniel Terdiman. Microsoft’s project natal: What does it meanfor game industry? CNET, 2009. URL http://news.cnet.com/

8301-10797_3-10253892-235.html.

[16] Anna Trifonova, Letizia Jaccheri, and Kristin Bergaust. Softwareengineering issues in interactive installation art. Int. J. Arts andTechnology, 1(1):43–65, 2008.