Mixed Interaction Spaces – expanding the interaction space with mobile devices Eva Eriksson, Thomas Riisgaard Hansen & Andreas Lykke-Olesen* Center for Interactive Spaces & Center for Pervasive Healthcare, ISIS Katrinebjerg, Department of Computer Science, University of Aarhus, Denmark. {evae},{ thomasr}@daimi.au.dk Department of Design*, Aarhus School of Architecture, Denmark alo@interactivespaces.net

Mobile phones are mainly interacted with through buttons, thumbwheels or pens. However, mobile devices are not just terminals into a virtual world; they are objects in a physical world. The concept of Mixed Interaction Spaces (MIXIS) expands the interaction with mobile phone into the physical world [Hansen et al. 2005]. MIXIS uses the camera in mobile devices to track a fixed-point and thereby establishes a 3 dimensional interaction space wherein the position and rotation of the phone can be tracked. In this paper we demonstrate that MIXIS opens up for new flexible ways of interacting with mobile devices. We present a set of novel, flexible applications built with MIXIS and we show that MIXIS is a feasible way of interacting with mobile devices by evaluating a MIXIS application against a traditional mobile interface. Finally, we discuss some design issues with MIXIS. Keywords: Mixed information spaces, mixed reality, mobile HCI, zoomable interfaces, mobile computing, spatial aware displays, drawable interfaces, gesture interaction.

1 Introduction Mobile devices such as mobile phones and PDA’s have been adopted into our daily life. Researchers at Nokia have observed that an important factor contributing to this is the personalization of the device, not just the communication possibilities

2 Anonymous

[Vänänen-Vaino-Mattila et al. 2000]. In constant use the mobile device becomes a personal object to such extent that it intensifies the user’s feeling of being inseparable from this unique thing. Still, the mobile devices are more and more becoming a personal computer in both functionality and interaction. The most common interaction is through buttons, thumbwheel or pen, and through something that can be characterized as a downscaling of the classic WIMP interface. The mapping of navigation and functionality to buttons, wheels and icons is not flexible and with low degrees of customization. The standard technique to view a large picture or map is scrolling by repeatedly press a button, roll a thumbwheel or drag a pen, and it is impossible to combine the manoeuvre with zoom, since the user has to divert the attention switching button to change function.

Designing for small mobile devices involves the classical problems of limited screen space, mapping functionality to small multifunctional buttons and traditionally a 2D interface. These problems can be reduced by expanding the interaction space outside the limits of the screen and the physical frames, and by using natural body gestures, the interface combine the digital and the physical world in a new 3D interaction space. By transforming the interface of the device into a 3D object it becomes a space belonging to the real world instead of the digital, and therefore reduces the cognitive load on the user 1.1 The concept of Mixed Interaction spaces In this paper we present a set of applications that expand the classical interface and interaction of the mobile device, to create a more natural interaction with a mixed reality interface. The applications are built on mixed interaction space [Hansen et al 2005], and demonstrate a new way to interact with digital information by using the existing camera of a mobile device to extract location and rotation of the device. Independent of the applications, the concept is to expand the interface of the mobile device outside the display by using the space between the front of the camera and a fixed-point, as illustrated in Figure 1. The space becomes the interaction space for gesture recognition. Moving the phone in the interaction space can be mapped to actions in the graphical user interface shown in the display or an action on a nearby device or display.

Figure 1: Diagram of the Mixed Interaction Space

To interact with the system the user only need one hand for the mobile device,

and then use the natural gestures of the body to address the system. Depending on

Title of a Full Paper 3

the application the device can be seen as having one to four degrees of freedom [Beaudouin-Lafon 2000]. Figure 2 displays how a four degree of freedom device can be generated by tracking the position and rotation of the device.

Figure 2: Diagram of gestures for interaction The size of the interaction space sets the borders both for the gesture recognition input and for the augmented interface, and is dependent on the size of the circle symbol representing the fixed-point and its distance from the viewpoint of the camera. A larger symbol spans a larger interaction space and therefore the gestures can be coarser. The fact that there is no fixed size opens up for the possibility to have small mixed interaction spaces, where the user have to use fine motor coordination or large spaces that requires the user to use larger movement.

The symbol can be anything as long as the camera can detect it. In the implemented concept a circle is used, it can be drawn or be a part of a decoration of some type and it can consist of different colours. Choosing simple symbols and using tolerant detection algorithms opens up for the possibility of drawable interfaces. The symbol can also be associated with a unique id, and combined with some type of generic protocol to send information, the concept can be used for controlling pervasive devices in the environment.

Even though the interaction is based upon natural body gestures, the concept does not require external sensor technology or specialized hardware. The concept can be implemented on standard mobile phones or PDA’s equipped with a camera.

The applications presented in this paper are built upon the principles of direct manipulation [Norman 1999], the actions are rapid, incremental and reversible and whose effect on the object is visible immediately. The users are able to act through gesturing and the display feedback or device functionality occurs immediately which convey the sense of causality.

In this paper we will demonstrate that MIXIS is a new and flexible concept for interacting with mobile devices that combines some of the properties of tangible interfaces with traditional mobile device interaction. We will argue for the novelty and flexibility of the concept by presenting four applications build with the concept. We have discussed several of the applications at small workshops, and we have made a formal evaluation of one of the applications to investigate and demonstrate that MIXIS is also a feasible way of interacting with mobile devices. Finally, we will discuss mapping and identity; two central aspects of MIXIS.

4 Anonymous

2 Related Work

Beaudouin-Lafon claims that it is becoming more important to focus on designing interaction rather than interfaces [Beaudouin-Lafon 2004]. Inspired by that, we argue that our applications are new compared to related work because: 1) support a high degree of mobility in the sense that it is not depending on any external tracking hardware, 2) are highly flexible because a wide set of different applications can be built by using the mixed interaction space in different ways and 3) provide a natural mapping between gestures and the interface since we are able to get quite precise information about the position of the mobile device in 4 dimensions.

2.1 New interaction techniques for mobile devices Several projects have explored different new interaction techniques for mobile devices [Fitzmaurice et al. 1993, Yee 2003, Patridge et al. 2002, Fällman et al. 2004, Masui et al. 2004]. Fitzmaurice [Fitzmaurice et al. 1993] uses a 6D input device to move around in a virtual world, Yee [Yee 2003] uses special hardware from Sony to track a PDA and interact with different applications using 3 dimensions and Patridge et al. [Patridge et al. 2002] have equipped a small portable device with tilt sensors for text entries. These systems use specialized tracking hardware that limits the mobility [Fitzmaurice et al. 1993, Yee 2003, Masui et al 2004] or tracks the device in just two dimensions [Fällman et al 2004, Yee et al. 2003, Masui et al. 2004], constraining the flexibility of the systems.

Accelerometers, can interact with an application by using tilting, rotation and movement of the device as input. The clear advantage of this interaction technique is its independence of the surroundings why it supports mobility very well. It supports new ways of interacting with applications e.g. scrolling in applications by tilting the device [Harrison et al. 1998].

2.2 Using cameras with mobile systems Other projects have experimented with using the camera on mobile devices for tracking and augmenting reality [Rekimoto et al. 2000, Rohs 2004, SemaCode, SpotCode]. Several of these projects aim at augmenting reality by using bar codes in the environment to impose a 3D digital image on reality [Rekimoto et al. 2000] and do not focus on the interaction. SemaCode [SemaCode] is focusing on how to bridge the gap between digital and physical material. SpotCode [SpotCode] and Rohs [Rohs 2004] focus on the interaction, but both systems relies on tracking two dimensional barcode and e.g. not on drawable symbols.

Interaction techniques that use integrated cameras strongly resemble interactions that can be designed with accelerometers. The movement, rotation and tilting of the device, can partly be extracted from running optical flow algorithms on the camera images. However, the camera images can provide more information than the movement, tilting or rotation vector. It can be used to identify a fixed point, and it can calculate its relative rotation, tilting and position according to this point.

Title of a Full Paper 5

2.3 Physical interfaces Mixed interaction space is related to tangible interfaces in the sense that both interaction techniques try to bridge the physical with the digital [Ishii et al. 1997]. In tangible interfaces the focus is on hiding the computer and having the interaction mainly in the physical world. This open up for highly intuitive interfaces like “The marble answering machine” [Ishii et al. 1997], but tangible interfaces are not that suitable for more advanced interfaces with a lot of functionality, because each object or function in the program would have to be associated with a physical representation. Mixed interaction space uses a combination of the physical world and the digital world. Most of the interaction possibilities are presented in the digital world, but to guide the interaction and to build shortcuts in the navigation a fixed-point is used in the real world.

3 Applications

3.1 Implementation Based on the conceptual discussion we designed and implemented a component to track the position and rotation of a mobile device within the mixed interaction space and identify a symbol drawn in the centre of the circle. Thereafter four applications based on the concept were implemented.

One of our main design goal was to build a system that everyone could use anywhere without having to acquire any new kind of hardware. Using the camera of mobile devices to track a fixed point fulfilled our requirements.

A circle is chosen as fixed-point in our prototype implementation of MIXIS, and it is appropriated for several reasons: 1) It is a symbol most people recognize and are able to draw. 2) There exists a lightweight algorithm for finding a circle in a picture. 3) The radius of the circle provides information about the distance between the camera and the circle. 4) The circle is suitable as a frame for different icons.

To detect the circle, we implemented the Randomized Hough Circle Detection Algorithm as described by Xu [Xu et al. 1990] on the phone. The main reason for choosing the randomized version is that it is lightweight and much faster than the Non-Randomized Hough Algorithm [Kälviäinen 1995]. We optimized the algorithm for the specific use by e.g. looking for only one circle in the picture.

The system is implemented in C++ for Symbian OS 7.0s on a Nokia 7610 mobile phone. To keep the interaction fluent and to reduce the memory used, we capture video in a resolution of 160x120 pixels in most of the prototype applications. In some of the applications where an instant response from the program was not required we used 320x240 pixels.

In the current implementation a black circle on a mainly non-black surface is tracked. The circle does not have to be perfect, the algorithm easily recognizes a hand drawn circle and the algorithm is also able to find the circle in different light conditions, which makes it more robust for use in different environments. Figure 3 demonstrates how the applications use the generic component.

6 Anonymous

Figure 3: Diagram of the system and how the applications use the generic

component. Depending on what application, the communication model is used to communicate with external devices.

3.2 Applications We have implemented four applications that use the mixed interaction space concept. To test the feasibility of the concept we carried out a formal evaluation of one of the applications and a set of workshops discussing some of the other applications. The conclusions from the evaluation are presented in the next section.

Figure 4: MIXIS applications (a – top left) ImageZoomViewer in use. (b – top right) Diagram of the LayeredPieMenu application (c – bottom left) DrawME, “Call Andy?” “no” – left, “yes” – right. (d – bottom right) DROZO in use on a wall display.

Title of a Full Paper 7

3.2.1 ImageZoomViewer The first application allows the user to pan and zoom simultaneously on a picture by moving the phone in the mixed interaction space, see Figure 1. When moving the phone closer to or further away from the circle the application zoom in and out on the image. Moving the phone to the left – right or up – down makes the application pan the image in the direction the phone moved.

We have worked with a basic scenario; navigation on a map. Maps are typically something people need when they are mobile, however, they are normally to large to fit on the screen of a mobile device. The user simultaneously needs both an overview of the entire map and details like street names. In Figure 4a we demonstrate the use of the ImageZoomViewer for browsing a subway map, here using a printed circle placed on a wall. The arrow points at the visual cue displayed on top of the map that indicated what kind of interaction the user was performing. In the picture the visual cue on the display shows that the user has placed the physical circle slightly to the right of the centre of the camera view why the visible area of the map is panning slowly to the left. The applications resembles the application implemented by [Yee 2003, Fällman et al. 2004], but in our application no specialized tracking equipment is used and we were able to both pan and zoom at the same time. 3.2.2 LayeredPieMenu In the application called LayeredPieMenus MIXIS is investigated and used to navigate a menu structure. The interaction space can contain numerous menus organized as pie menus [Callahen et al. 1998] on top of each other. When the camera recognizes the circle a pie menu appears and augments the circle on the display. The pie menu consists of up to eight function segments that surround an info text explaining which menu is at hand. The functions in each menu can be selected by panning the phone towards the specific segments and back to the centre. By making a simple gesture towards the circle and back again the next menu is selected and moving the phone away from the circle and back again selects the previous menu. The diagram in Figure 4b demonstrates the principle of the LayeredPieMenu application where virtual pie menus are stacked on top of each other. 3.2.3 DrawME In the DrawME application the device is, besides from recognizing the clean circle, also able to distinguish between a set of hand drawn symbols within the circle. Like in [Landay et al. 2001] DrawME opens up for the idea of drawable interfaces where the user is able to draw shortcuts, to applications in the real world e.g. on paper, whiteboards and walls. In a sense the user add another layer or functionality to disposable doodling. When the user draws a circle containing a specific symbol the camera recognizes the input and performs the function mapped to the specific symbol. The algorithm stores a set of masks of known symbols and finds the best match between the symbol in the centre of the circle and the known masks. At the moment the mask is hard-coded to the different symbols, but we are working on a user interface for creating and mapping new symbols. In DrawME we mapped

8 Anonymous

different symbols to the single function of calling a certain contact from the address book illustrated in Figure 4c. To either confirm or reject calling the contact appearing on the display the user pan towards the yes and no icons displayed on the phone interface. 3.2.4 DROZO The application Drag, Rotate and Zoom (DROZO) focus on how the mobile device can be used to interact with pervasive devices in the surroundings equipped with an interactive circle. The commands are sent through a generic protocol, see Figure 3. We enhanced the application by putting a circle underneath an x-ray picture on a large wall display, allowing the user to drag the picture around on the screen using the mobile device. The user is able to zoom in and out on the picture by moving the device closer to or away from the display, and to rotate the picture by rotating the phone. In our first prototype we used GPRS to communicate between the wall and the phone, but in the new version we use Bluetooth to communicate between the device and the screen. To be able to rotate the picture we added a small mark to the circle that allowed us to detect rotation as illustrated in figure 4d.

4 Evaluation

Our main purpose of introducing the MIXIS concept is not to argue that this is necessary a faster way to interact with mobile devices: Our main purpose is to show an alternative and more flexible interaction concept. With the ImageZoomViwer we performed a usability test with fifteen persons to see if it is feasible to use MIXIS as an interaction technique. We have had some preliminary experiences with some of the other applications at a workshop where we invited a group of users and their children to test some of the applications. However, in this paper we will focus mainly on the usability test of ImageZoomViewer. 4.1 Usability test of ImageZoomViewer We wanted to investigate if users were able to use our interface as efficient as the traditional interface offered by mobile devices, to use the result as guidelines for further development. Therefore a usability study was conducted, comparing the ImageZoomViewer application to a standard application for picture viewing from Nokia. An even more important aspect was to test if MIXIS was perceived as a fun complement to traditional interaction techniques. The participants were 15 in total, and they had various degrees of experience from mobile devices, spanning from not owning one to software developers for mobile phones. None of them had ever before seen or used gesture interaction for mobile devices.

The test was performed in a quiet conference room, a Nokia 7610 mobile phone was used, and there was a drawn circle on a white paper on the table. The two tasks were designed to test map viewing, a typical use case for mobile devices, including shifting degrees of zoom for overview and detail. For each of the two tasks, a conventional Nokia interface for image viewing using buttons was compared to the ImageZoomViewer application. Each participant did both tasks using both interfaces, where half of the participants started out with the conventional interface

Title of a Full Paper 9

and half with the new interface and then switched for the second task. Before starting instructions were given in both techniques and both interfaces were practiced on a dummy data set for a few minutes before proceeding with timing tasks. For each task a new data set was used, to reduce learning effects. The order in which the different data sets were used changed for half of the test group. 4.1.1 Task 1 First application: Given a subway map, locate the blue line and follow it from the most southern end station to the most northern end station of that line. Read the names of the end stations out loud.

Second application: Locate the green line and follow it from the most southern to the most northern station. Read the names of the end stations out loud. 4.1.2 Task 2 Second application: Given a second subway map, locate a station in the centre of the map and tell out loud the colour of all the lines that stop there. Follow one of those lines to the two end stations and tell the name of those. First application: Go to a different centre station and tell what lines stop there. Follow one of those lines to its both end stations, and tell the names of the end stations out loud. 4.2 Result of usability test with ImageZoomViewer Independent of what data set or interface, the user error rates were not significant, and there was no difference between the two data sets for each task. After the test was over, the participants were asked which application they preferred. The majority of the test persons, 80%, strongly preferred ImageZoomViewer for map viewing. Table 1 and 2 presents a summary of the experimental data.

The conventional interface was 6% faster then ImageZoomViewer in the first test, but in the second test the ImageZoomViewer was 9% faster, as illustrated in Table 1. These results show that gesture interaction with ImageZoomViewer is a quicker method the second time, concluding that with some practice the concept is actually a more effective navigational technique.

Table 1: Experimental data from the usability test where ImageZoomViewer was

tested against a conventional Nokia interface for viewing pictures. The bars represent the time to complete two tasks (T1 and T2) for each interface.

Table 2: Subjective preferences from the usability test.

10 Anonymous

During the user tests, it became obvious that the distance between the camera on the mobile device and the circle on the object was very relevant. The female test persons were a bit shorter in height, and the positioning of a circle on the table made the phone end up closer to the face leading to that the interaction was not natural to the same extent as for the men. It was a lack in our test that the test persons were not asked to test different positions of both the circle and of themselves, to find the most comfortable and effective distance.

The most positive comments were about the direct connection between the physical movement and the interface, and also the possibility to pan and zoom simultaneously. The overall experience was that it was intuitive, fun and effective. The most frequent complaint concerned the refresh rate and the sensitiveness of the system. This problem was due to the size of the circle: we should have chosen a larger circle, since enlarging the circle also enlarges the span of the interaction space and therefore the gestures. The ImageZoomViewer was due to the sensitiveness considered a bit less precise than the conventional interface. In some cases there were comments about the small size of the letters, which was a problem due to the quality of the picture we had chosen. 5 Discussion The main outputs from the tracking component are the location and rotation of the device in relation to the fixed-point and in some cases information about the symbol inside the circle. Applications can use this information in a number of ways to interact with the device. This flexibility open up for the creation of a wide variety of different types of applications as shown above. We found two aspects relevant in describing the characteristics of the different application. The first was how the movement of the phone in the mixed interaction space was mapped to the application and the second was if the tracked fixed-point was associated with an identity or ID. Below we will thoroughly discuss these two aspects. 5.1 Mapping applications to the Mixed Information Space Basically two different types of mapping were found present in the applications we explored, natural and semantic mapping. 5.1.1 Natural mapping In the first type of applications we tried to make a tight coupling between the physical movement and the application, trying to accomplish natural mapping introduced by Norman [Norman 1999]. One example of this is in the ImageZoomViewer application, where moving the device to the left, right, up or down makes the application pan the image. Moving the phone closer or further away from the circle the application zoom in and out. Another example is the DROZO application that uses the rotation of the phone to rotate the current picture.

Title of a Full Paper 11

Figure 5: Diagram of the stable zone in relation to the drawn circle.

To further discuss mapping we need to introduce a distinction between absolute

and relative mapping. In absolute mapping there exists a one to one mapping between a specific position in the mixed interaction space and the application. E.g. each time the phone is in a specific position in the space the application will scroll and zoom to the same position. The project suggested by Yee uses what we call absolute mapping [Yee 2003].

Relative mapping maps a specific position in the space to a movement vector instead of a position. Keeping the device in the centre of the mixed interaction space resembles the movement vector null, which we call the stable zone illustrated in figure 5. If the device is moved outside the stable zone the position of the device is mapped to a movement vector in the application. E.g. moving the device to the left of the stable zone would be mapped to keep scrolling to the left until the device is moved back into the stable zone. The further away the device is moved from the stable zone the faster the application scrolls. The project suggested by Fällman uses relative mapping [Fällman et al. 2004].

We explored both relative and absolute mapping in e.g. the ImageZoomViewer application. With absolute mapping moving the phone towards the circle results in a zoomed in picture, moving the phone to the left edge of the space moves the focus to the left edge of the picture and so on. One of the problems with absolute mapping is that the Mixed Interaction space has the form of an inversed pyramid (see figure 1), meaning that, if the device is close to the fixed-point, the x, y plane is smaller than when the device is far from the fixed-point. This property makes mixed interaction space unsuitable for absolute mapping or at least absolute mapping on all three axes. It is still possible to use absolute mapping for instance for zooming and then use relative mapping for panning. We found two other problems with absolute mapping. The image captured by the camera has to have similar size as the picture being watched; otherwise a small movement with the device will make the picture jump several pixels. Secondly, because the mechanism for determining the exact position and radius of the circle is not always exact, the picture becomes more vivid than with relative mapping.

Relative mapping was best suited in our applications. As an example, using a circle with a diameter about 2,5cm made a stable zone approximately 10 cm above the circle as illustrated in figure 5. When the device is within this zone the picture is fixed and when moving the phone forward towards the circle or away from the circle the picture is zoomed in or out with a speed relative to the distance from the stable zone. The same applies for panning. The disadvantage with relative mapping

12 Anonymous

is that it does not provide the same spatial awareness as absolute mapping about the position on the picture. Relative mapping was used in the evaluated applications. 5.1.2 Semantic mapping The second type of mapping we use is what we call semantic mapping. With semantic mapping moving the phone in a specific direction does not necessarily map to the application moving in the same direction. With semantic mapping a metaphor is used to bridge between the physical movement and the action on the device. For instance moving the phone to the left might correspond to the action play media file and not to move left. This kind of mapping resembles the mapping used in gesture based application where performing a gesture is mapped to a specific function and not the same movement in the interface.

A characteristic of semantic mapping is that it is discrete; the space is divided into different zones that can be mapped to activate different functions. E.g. in the LayeredPieMenu moving the phone down towards the fixed-point and into the stable zone is mapped to the function “go to the next menu”. The semantic mapping between the gesture in the interaction space and the application can be arbitrary which also results in problems with purely gesture based interfaces. How are the gestures the system recognizes visualized and how are these gestures mapped to the different applications? With LayeredPieMenu we use the display of the mobile device to guide the user. By graphically visualize the different menu items in the display the user was helped figuring out e.g. that making a gesture to the left would activate the function displayed to the left on the screen. 5.2 Mixed Interaction Space with or without Identity One of the main strengths we found of Mixed Interaction Space in comparison to other systems [Rohs 2004, Semacode, Spotcode] is that the system also works with simple symbols e.g. a circle drawn by hand. We found, that a set of very different applications could be designed by giving the circle different types of identity. We made a distinction between interfaces needing solely a simple circle to function (simple fixed-point interfaces), interfaces that uses a simple fixed-point with an associated icon drawn by hand (drawable interfaces) and interfaces that need to associate a unique ID with the fixed-point (identity interfaces). 5.2.1 Simple Fixed-Point Interfaces The simple circle interface proved to be the most flexible. A simple interface just needs to have the software to recognize a circle to work. The circle could be drawn with a pen, but we also explored how to use different things as a marker like special finger rings or a black watch. The ImageZoomViewer and the LayeredPieMenu are examples of simple interfaces. 5.2.2 Drawable Interfaces The main characteristic of drawable interfaces is that the system is can recognize different symbols drawn by hand within the circle and provide a set of different mixed interaction spaces on top of each circle, as illustrated in DrawME. In

Title of a Full Paper 13

[Landay et al. 2001] Landay present an application that recognizes the widget in a hand drawn interface. We wish to pursue the possibility with drawable interfaces, but in contrast to Landay in our system the drawing is the actual interface.

Instead of trying to squeeze a lot of functionality into a single device, drawable interfaces are able to customize the interface with only the functionality required in the given situation. The drawn symbols can be seen as a kind of physical shortcuts into the digital world. In this way, drawable interfaces resemble tangible interfaces that also try to distribute the controls to the real world. One of the problems with tangible interfaces as pointed out by [Greenberg 2002] is that you have to carry a lot of special tangible objects with you if you want to use these interfaces in a mobile setting. Greenberg [Greenberg 2002] propose to use easily customizable tangible objects, but still you have to use a set of tangible objects. With drawable interfaces all you need is a piece of paper or a white board and a set of pens, and after finishing with the interface it can be wiped out or thrown away.

Another advantage with drawable interfaces is that each circle can be associated with a 4D mixed interaction space with the interaction possibilities demonstrated in for instance the ImageZoomViewer. Furthermore this application could be combined with the LayeredPieMenu concept as a fast physical shortcut to certain predefined functions in the phone. This could e.g. be a LayeredPieMenu containing the four most called persons, send/receive mail, and toggle sound and so on.

The number of symbols the system recognizes and tracks is dependent on the software, the hardware and the context. Sometimes it is difficult for the application to recognize a colour because the colour seen by the camera depends on the quality of the camera, the lightning, the pen used to draw the colour, and the surface. Therefore a small set of different colours are best suited for drawing the symbols. The same restriction applies for symbols. Because the symbols are hand drawn and not computer generated to symbols never looks exactly the same. Choosing a set of symbols that does not resemble each other works best with drawable applications.

Drawable interfaces opens up for a whole new area of customization and personalization of the interface of the mobile device, which is one important factor contributing to the success of mobile devices. The user is able to adjust the device to recognize new and personal symbols, to make it even more “intelligent” and unique, since the user becomes the interface designer. In the workshop with DrawME, the participants strongly welcomed this possibility to customization, both because it is fun and that it provides the ability to personalize their device. The workshop also taught us the importance of having the user in total control of the mapping, and not have automatic mapping of any kind. We consider that it should be fun to interact with technology, and especially with the mobile and personal devices. Schneiderman highlights this aspect with a recent question: “Did anyone notice that fun is part of functionality?” [Schneiderman 2004]. 5.2.3 Identity Interfaces In the final type of interfaces the fixed-point is associated with a specific identity or unique ID. The identity can be read by printing a barcode in the circle [Semacode], providing the identity by using short range Bluetooth [Blipnodes] or by RFID tags [Want 1999]. The corresponding mixed interaction space can then be

14 Anonymous

stored in the device, transmitted through for instance Bluetooth or downloaded from the internet. We used identity interfaces in the DROZO application.

Identity interfaces are especially suitable for interacting with external devices or as shortcuts to specific places on the internet. Using MIXIS to interact through identity interfaces can be seen as a possible method to interact with the “invisible computer”. When computers get smaller, embedded or even invisible it is becoming more difficult for the user to know how to interact with them. A circle on a wall can be used as a visual cue, signalizing the existence of a hidden MIXIS interface and can at the same time be used as fixed point for the interaction space. In this way, the context can be used to reduce interface complexity. 6. Conclusion The main contribution of this paper has been to introduce Mixed Interaction Spaces, a concept that investigate and demonstrate that the interaction with mobile devices is not something that has to be limited to the screen and buttons on the phone. By using the camera of a mobile device we are able to combine the phones abilities with the physical environment and introduce a new interaction concept.

In this paper the main focus has been to introduce MIXIS and demonstrate some novel applications with the concept. The applications use the camera in the mobile device to track a fixed point and thereby establish a 3 dimensional interaction space wherein the position and rotation of the device is calculated. The first application, ImageZoomViewer, allows the user to pan and zoom simultaneously on a picture by moving the phone in the mixed interaction space. In the application called LayeredPieMenu the mixed interaction space is used to navigate a layered menu structure. In the DrawME application the device is able to distinguish between a set of hand drawn symbols within the circle. The application Drag, Rotate and Zoom (DROZO) focus on how the mobile device can be used to interact with pervasive devices in the surroundings equipped with an interactive circle.

Mapping and identity, two central issues with MIXIS have been discussed and some relevant distinctions and design challenges have been pointed out. However, mapping and identity are just two aspects of MIXIS and we can see several other possibilities in combining tangible interfaces and mobile phones. Because the mobile phone is a highly personal device most people have we are e.g. currently looking into how to use the concept to design multi-user applications and so far MIXIS seems to have some interesting properties in this domain. 7. Acknowledgements The work has been supported by funding from Center for Interactive Spaces and Center for Pervasive Healthcare under ISIS Katrinebjerg at the University of Aarhus. We would like to thank people at Center for Interactive Spaces and Center for Pervasive Healthcare, especially Kaj Grønbæk and Jakob Bardram. We also wish to thank the people involved in user tests and workshops.

Title of a Full Paper 15

References Beaudouin-Lafon, M. Instrumental interaction: an interaction model for designing post-WIMP user interfaces. Proceedings of the SIGCHI conference on Human factors in computing systems. Netherlands (2000), 446 – 453. Beaudouin-Lafon, M. Designing Interaction, not Interfaces. AVI’04, Italy, (2004), 15-22. BlipNodes. http://www.blipsystems.com/ Callahen, J., Hopkins, D., Weiser, M., Shneiderman, B. An empirical comparison of pie vs. linear menus. Proceedings of the SIGCHI conference on Human Factors in computing systems. USA (1998), 95-100. Fällman, D. Lund, A. Wiberg, M. Scrollpad: tangible scrolling with mobile devices, In Proceedings of the 37th Annual Hawaii International Conference on System Sciences, (2004), 286- 291. Fitzmaurice, G. W., Zhai, S., Chignell, M.H. Virtual Reality for Palmtop Computers, In ACM Transactions on Information Systems,11, 3 (1993), 197-218. Greenberg, S., Boyle, M., Customizable Physical Interfaces for Interacting with Conventional Applications. In CHI Letters, 4, 2(2002), 31-40. Hansen, T.R., Eriksson, E., Lykke-Olesen, A. Mixed Interaction Space – Designing for Camera Based Interaction with Mobile Devices. In Proceedings of CHI 2005, ACM Press, (2005). Harrison, B.L., Fishkin, K.P., Gujar, A., Mochon, C.,Want, R. ‘Squeeze me, hold me, tilt me! An exploration of manipulative user interfaces’, in ACM CHI’98, ACM Press, Los Angeles, CA (1998). Ishii, H. and Ullmer, B. Tangible bits: towards seamless interfaces between people, bits and atoms, In Proceedings of the CHI 97, ACM Press, (1997), 234-241. Kälviäinen, H., Hirvonen, P., Xu, L., Oja, E. Probabilistic and non-probabilistic Hough transforms: overview and comparisons. In Image and Vision Computing 13, 4 (1995), 239-252 Landay, J., Myers, B. Sketching Interfaces: Toward More Human Interface Design, In Computer 34, 3 (2001), 56-64.

16 Anonymous

Masui, T., Tsukada, K., Siio, I. MouseField: A Simple and Versatile Input Device for Ubiquitous Computing, In Proceedings of UbiCom2004, Springer (2004), 319-328. Norman, D. The Design of Everyday Things, Doubleday, New York, (1999), p 23, Patridge, K., Chatterjee, S., Sazawal, V., Borriello, G., Want, R. TiltType: Accelerometer-Supported Text Entry for Very Small Devices, In CHI Letters, 4, 2, (2002), 201-204. Rekimoto, J., Ayatsuka, Y., CyberCode: designing augmented reality environments with visual tags, In Proceedings of DARE 2000 on Designing augmented reality environments (2000), 1-10. Rohs, M., Real-World Interaction with Camera-Phones, 2nd International Symposium on Ubiquitous Computing Systems (UCS 2004), 2004. Schneiderman, B. Designing for Fun: How can we design user interfaces to be more fun? Interactions Volume XI.5 (2004), 48-50. SemaCode. http://semacode.org/. SpotCode. http://www.highenergymagic.com/spotcode/. Vänänen-Vaino-Mattila, K., Ruuska, S. 2000. Designing mobile phones and communicators for consumers’ needs at Nokia. In Information Appliances and Beyond; Interaction Design for Consumer Products, E. Bergman, Ed. Morgan Kaufman, San Francisco, CA, 169-204. Want, R., Fishkin, K., Gujar, A., Harrison B., Bridging Physical and Virtual Worlds with Electronic Tags. In Proc. CHI 1999, ACM Press (1999), 370-377 Xu, L., Oja, E., Kultanen, P., A new curve detection method: Randomized Hough Transform (RHT). In Pattern Recognition Letters 11 (1990), 331-338 Yee, Ka-Ping. Peephole Displays: Pen Interaction on Spatially Aware Handheld Computers. In Proceedings of CHI 2003, ACM Press, (2003), 1-8.