Project ReportonAugmented Reality using Wiimote

Submitted in partial fulfillment of the requirementsof the degree ofBachelors of EngineeringbyAdil Khan (66)Huzefa Saifee (42)Roshan Shetty (49)

Guide:Prof. Rajan S. Deshmukh

Computer Engineering DepartmentRizvi College of Engineering

University of Mumbai2014-2015

CERTIFICATE This is to certify that the project report entitled Augmented Reality using Wiimote is a bonafide work of Adil Khan (66), Huzefa Saifee (42) and Roshan Shetty (49) submitted to the University of Mumbai in partial fulfillment of the requirement for the award of the degree of Bachelor of Engineering in Computer Engineering.

(Prof. Rajan S. Deshmukh) Guide

(Prof. Dinesh Deore)(Dr. Varsha Shah)Head of Department Principal

Project Report Approval for B.E.This project report entitled Augmented Reality using Wiimote by Adil Khan, Huzefa Saifee and Roshan Shetty, is approved for the degree of Bachelor of Computer Engineering.

Examiners

1.---------------------------------------------

2.---------------------------------------------

Guide

1.---------------------------------------------

2.---------------------------------------------

Date: 20th April, 2015

Place: Mumbai

DeclarationI declare that this written submission represents my ideas in my own words and where others' ideas or words have been included, I have adequately cited and referenced the original sources. I also declare that I have adhered to all principles of academic honesty and integrity and have not misrepresented or fabricated or falsified any idea/data/fact/source in my submission. I understand that any violation of the above will be cause for disciplinary action by the Institute and can also evoke penal action from the sources which have thus not been properly cited or from whom proper permission has not been taken when needed.

-----------------------------------------Adil Khan

-----------------------------------------Huzefa Saifee

----------------------------------------- Roshan Shetty

Date: 20th April, 2015

ABSTRACT

The goal of this project is to create a customizable three dimensional virtual reality display on a system available to any non-technical user. This system will use the infrared camera component of a standard Nintendo Wiimote to track a users head motions in the left, right, forward, backwards, up and down directions. The virtual display will be a customizable image projected onto a screen or simply shown on a computer or TV monitor. In order to make the two dimensional image appear three dimensional, the image on the display will continually change according to the position of the users head. As the user moves their head to the left and right, potions of the image displayed will be shifted to the right and left respectively by different amounts depending on their depth location in the image. Likewise, as the user moves their head closer and further from the display, portions of the image will be increased or decreased in size again depending on their depth location. In this way the image is being continually redrawn to match the users perspective or vantage point. The user will be able to select a number of images of their choosing and place them in the virtual display assigning them each a position in space based on x, y, z coordinates and relative scale. The system will create a continually augmented display according to the motion and location of the users head to create a three dimensional presentation on a two dimensional screen.

Index

Sr. NoTitlePage No

1.Introduction1

2.Review and Literature 2

2.1.Paper 13

2.2.Paper 24

3.Proposed Methodology5

3.1Section6

3.1.1.Subsection7

4.Conclusion9

5.References10

Appendix11

Acknowledgement12

List of Figures

Sr. NoTitlePage No

1.1.Overall System Diagram

1.2The setup used to perform head tracking with the Wiimote and IR LEDs.

1.3 The change in the visible field when the user moves to a different angle relative to (a) a window and (b) a monitor.

1.4

3.1

3.2 The change in the visible field when the user moves closer to (a) a window and (b) a monitor

Schematic Block Diagram of Apparatus

Block Diagram Interfacing Wiimote.

.

List of Tables

Sr. NoTitlePage No

1.1.Table 11

Chapter 1Introduction

The goal of this project is to take a three dimensional virtual reality space and make it customizable to the user. This virtual reality space will consist of a two dimensional image that is continually redrawn to rearrange the size and location of individual images depending on the users changing perspective of the scene. The user will be able to take a background image and insert a variety of different images at different locations of the foreground to create their own virtual display. Each image placed in the foreground, will have its own x, y, z coordinates in the three dimensional space and a specific size relative to all other objects contained in the space. Each aspect of the images location and even the images themselves will be determined by the user in a closed loop system so that values may be changed without the need to recompile and restart the system. The three dimensional characteristic of the space will be conveyed through use of head tracking with a Wiimote. The user will wear a specially designed hat or set of glasses with infrared LEDs embedded on the front facing towards the screen. A Wiimote will be placed under or above the screen facing the user so that the Wiimotes infrared camera can determine location and movements of the LEDs. As the user moves their head and the LEDs change location, the information is sent to the computer via a Bluetooth connection. The computer then parses the data from the Wiimote to determine the changing orientation of the users head. The system uses the head tracking information to perform an appropriate matrix transformation on the three dimensional virtual space. As the user moves their head to the left, the images are moved to the right by different amounts based on their specified depth in the space. If the user moves closer or farther away from the screen, the images will be scaled larger or smaller, again by amounts determined by their depth in the space, to simulate a three dimensional environment. A communication block diagram for the overall system is shown below in Figure 1.

Figure 1.1 - Overall System Diagram

In this report we will discuss the background of the project and why it was deemed important enough to invest time into, the specific requirements of the end product, our design, construction and testing of the system, and finally what we gained from working on this project and our recommendations for future development of the system.

Figure 1.2: The setup used to perform head tracking with the Wiimote and IR LEDs.

In line with the growth of the game industry, the demands for realism in modern computer games increase as well. Realism is often raised through improved graphics, articial intelligence, sound eects and similar. The player interaction devices are however, mostly limited to the use of mouse, keyboard and conventional game controllers. The possibilities have improved somewhat by the introduction of Wii1, Nintendos newest game console which was introduced in 2006. Now players can interact more directly and naturally with the games, thus achieving higher realism. This is done through the Wiimote controller which, among other things, features motion-sensing. However, critical areas are still left behind. There is for example poor access to equipment allowing the player to change the view perspective by moving her head. This can indeed already be achieved using available virtual reality (VR) equipment, but for most users it is not within an acceptable price range.It turns out that equipment allowing this kind of interaction does not necessarily have to be expensive. It can be achieved using an infrared (IR) camera and some light emitting diodes (LEDs). By placing the camera by the monitor and the LEDs by the head, the placement of the LEDs can be found, hereby allowing us to determine the users movement and changing the eld of view according to this. This yields the illusion that the monitor is like a physical window into another room instead of just a static photo. This can increase the realism for especially 3D games dramatically.Johnny Lee [14] from Carnegie Mellon University has realized the task using the IR camera on the Wiimote to facilitate the job using a simple setup illustrated in gure 1. Normally registration of images is a dicult job and forms an entire research eld, but when taking advantage of the build in IR camera and image processor of the Wiimote, it becomes quite simple to detect and track numerous positions. Furthermore the Bluetooth support and low price of the controller makes it easily accessible. Based on these arguments, we feel that the idea presented by Johnny Lee calls for further investigation. In this paper we will therefore try to mimic his work and determine the eld of view in a 3D game like world from the available data. Through a user study we furthermore wish to evaluate whether our solution is suited for interaction in a 3D world.To keep the focus of the project on the subject at hand, we will assume that the reader is familiar with the process of setting up a camera in a 3D scene. Thus terms like the up vector and projection matrix will not be explained in the paper. A thorough introduction to this subject can be found in [4].Included with the paper is a CD-ROM containing a digital copy of the paper, the source code, application program interface (API) documentation and a video illustrating our solution in action.

Wiimote

In November 2006, Nintendo released its fifth home videogame console, the Nintendo Wii. The companys previous game console, the Gamecube, hadnt fared well in terms of market share against the much higher-powered alternatives released by its competitors, Microsoft and Sony. At first the Wii also seemed significantly underpowered relative to its competitors. However, one year later it became the market leader of its console generation, selling over 20 million units worldwide.1 This success is largely attributable to the innovative interactive technology and game-play capabilities introduced by the consoles game controller, the Wii remote, shown in Figure 1. The Nintendo Wii remote, or Wiimote, is a handheld device resembling a television remote, but in addition to buttons, it contains a 3-axis accelerometer, a high-resolution highspeed IR camera, a speaker, a vibration motor, and wireless Bluetooth connectivity. This technology makes the Wii remote one of the most sophisticated PC-compatible input devices available today; together with the game consoles market success, its also one of most common. At a suggested retail price of US$40, the Wii remote is an impressively cost-effective and capable platform for exploring interaction research. Software applications developed for it have the additional advantage of being readily usable by millions of individuals around the world who already own the hardware. Ive recently begun using Internet video tutorials to demonstrate interaction techniques supported or enabled by the Wii remote. In just a few weeks, these tutorials have received over six million unique views and generated over 700,000 software downloads. In this article, I will talk about the Wii remotes technology, cover whats involved in developing custom applications, describe intended and unintended interaction techniques, and outline additional uses of the device.The Wiimote is a controller for the Wii. In contrast to most other controllers for game consoles, the input methods are not just buttons and analogue sticks, but an IR camera and motion-sensing. In this section we will describe these features. The technical specification is based on . The appearance of the Wiimote is designed much like a TV remote as seen in figure below. It has 12 buttons, where 11 are on the top and one is located underneath. The Wiimote has 16 kilobyte EEPROM2 of which a part is freely accessible and another is reserved. The Wiimote also houses a speaker and the top has holes which the sound can come out of. Four blue LEDs are on the top. If the Wiimote connects to a Wii the LEDs first indicate the battery level and afterwards which number the Wiimote is connected as. A small rotating motor is placed in the Wiimote which can make the Wiimote vibrate. A plug is located in the end of the Wiimote. Through this plug attachments can be connected. A couple of peripherals have been released.

However, the most used is the Nunchuck. The Nunchuck is a device with two buttons, an analogue stick and motion-sensing as the Wiimote itself. The Wiimote can detect motion in all 3 dimensions. This is achieved through accelerometers inside. For a thorough discussion of the functionality. The front of the Wiimote has a small black area like a normal TV remote. The difference is that a TV remote has an IR LED beneath, where the Wiimote has an IR camera. By placing a so called sensor bar (essentially IR LEDs powered by the Wii) below the TV, the Wiimote can be used as a pointing device for the Wii. As can be seen on Figure 3 the LEDs in the sensor bar are located with space between them. This makes it possible to calculate the relative position of the Wiimote with regards to the sensor bar. When using the Wiimote as a pointing device for the Wii, one doesnt actually point at things on the TV, but is pointing relative to the TV. To minimize the amount of data being transferred from the Wiimote to the Wii, it does not actually transfer every image taken by the Wiimote. Insteadthe Wiimote calculates the position of each point and sends the x- and y-coordinate for each of them together. It can also send other information such as a rough size of the points. It is noted that the LEDs that can be seen of Figure 3 are taken as two separate LEDs and not 10. This is because they are located so close to each other. The Wiimote can register up to four separate points at a time.

The Wiimote does not just take an image, analyze it and send the result of the positions to the Wii, it also keeps track of the points. Every time a point is detected it is assigned a number from one to four. If for instance two points are detected and the one marked as 1 is moved out of range, the other point is still marked as 2. If the point is then moved within range again it is marked as 1, but the Wiimote of course can not tell if it is the same LED as before or a new one.

Inside the Wii remoteAlthough the Wii remotes official specifications are unpublished, the global hacking community has collectively reverse-engineered a significant portion of the technical information regarding its internal workings. Much of this work has been collected in online wikis at http://wiili.org and http://wiibrew.org. The body of knowledge at these sites represents contributions from numerous individuals and constitutes the source for most of the information presented in this section. Because many low-level details are available online and, furthermore, are likely to be refined and updated as more information is uncovered, the following descriptions of each major Wii remote component represent only higher-level details relevant to building custom applications.

Infrared Camera Tracker: In the tip of each Wii remote is an IR camera sensor manufactured by PixArt Imaging, shown in Figure 2. The camera chip features an integrated multiobject tracking (MOT) engine, which provides high-resolution, high-speed tracking of up to four simultaneous IR light sources. The camera sensors exact specifications are unpublished, but it appears to provide location data with a resolution of 1,024 768 pixels, more than 4 bits of dot size or light intensity, a 100 Hz refresh rate, and a 45 degree horizontal field of view. The integrated hardware object tracking minimizes the data transmitted over the wireless connection and greatly simplifies the implementation of camera-based tracking applications. These specifications outperform comparably priced webcams, which typically provide 640 480 tracking at 30 Hz. Webcams also require significant CPU power to perform real-time computer-vision tracking. Specialized IR camera trackers, such as those from Natural Point Systems (www.naturalpoint. com), can provide 710 288 tracking at 120 Hz, but at a significantly higher cost of $180.

Accelerometer:Analog Devices manufactures the ADXL330, a 3-axis linear accelerometer that provides the Wii remotes motion-sensing capability. It has a +/3 g sensitivity range, 8 bits per axis, and a 100 Hz update rate.

Buttons:The Wii remote has 12 buttons. Four are arranged in a standard directionalpad layout. One button is on the bottom providing a trigger-like affordance for the index finger. The remaining seven buttons are intended to be used by the thumb. The remote design is symmetric, allowing use in either the left or right hand.

Vibration motor (tactile feedback):A small vibration motor provides tactile feedback. The motor is similar to those used in cell phones. The motor state has only binary control (on and off), but you can vary the feedback intensity by pulsing the motor activationthat is, by rapidly turning the motor on and off at different duty cycles.

Light-emitting diodes (visual feedback):Four blue LEDs at the bottom of the remote are typically used to indicate player IDs (1 to 4). Each LEDs state is individually addressable. Similarly to the vibration motor, pulsing the state creates varying levels of brightness.

Speaker (auditory feedback):A small speaker in the remotes center supports in-game sound effects and user feedback. The audio data streams directly from the host with 4-bit, 4 KHz sound similar in quality to a telephone.

Bluetooth connectivity:Communication runs over a wireless Bluetooth connection. The connection uses a Broadcom 2042 chip, which Broadcom designed for devices that conform to the Bluetooth Human Interface Device standard, such as keyboards and mice. The remote isnt 100 percent compliant with the HID standard, but it can connect to many Bluetooth-capable computers.

Internal Flash Memory:The onboard memory is approximately 5.5 Kbytes. Its used for adjusting the device settings, maintaining output state, and storing data. Nintendo designed it to let users transport and store a personal profile, called a Mii. This memory allows data and identity to be physically associated to a given remote.

Expansion Port:At the base of the remote is a proprietary six-pin connector used to communicate with and power extension controllers such as the Nintendo Nunchuk, Classic Controller, or a guitar controller. These extensions provide alternative form factors and additional input capabilities. The port provides 3.3 V of power and 400 KHz of I2C serial communication, to which a microcontroller can easily interface and effectively provide a Bluetooth-to-I2C bridge.

Batteries:The Wii remote uses two AA batteries and has an operating time between 20 and 40 hours, depending on the number of active components. Approximately 8 bits of battery-level resolution are available.

Wii Console Interaction Techniques

Wii users hold the remote controller in one hand and point it at a television that has a Wii sensor bar either above or below the screen. The term sensor bar is a misnomer because the device doesnt contain sensors; rather, it contains two groups of infrared LEDs. The Wii remotes IR camera sees the two groups and provides a method of laser pointer-style input. The software can transform the x, y coordinate dot pairs to provide an averaged x, y coordinate pair, a rotation, and a distance. The x, y, and rotation values correspond to the controllers yaw, pitch, and roll, respectively, and the distance is estimated using the known physical separation of the two IR groups and the cameras fixed field of view. Common Wii game interactions using the controller as a pointer include selection, navigation, aiming a weapon or tool, drawing, rotating objects, and push-pull interactions. Although the remote is frequently used as a pointer, no game currently makes a significant attempt to ensure that the cursors onscreen position accurately matches the screen planes intersection with the ray defined by the axis of the Wii remote. Assumptions are made regarding the screens visual angle and the scale of movement. However, this doesnt appear to have a significant impact on users pointing ability in most contexts. This might be because the game provides constant visual feedback of the cursor position, which lets users rely on relative movements rather than absolute aiming. Use of the accelerometer data within Wii games varies from basic shake triggering, to tilt-and-balance control, to simple gesture recognition. WiiSports, the mini-game that comes with the Wii console, might currently involve the most intricate use of the accelerometer data. WiiSports encourages players to swing the remote in the imaginary context of bowling, boxing, or playing tennis, baseball, or golf. The game appears to register subtle variations in swing dynamics and thus affect the simulation. Though the motion recognition might not necessarily be accurate, the experience is quite compelling. However, the majority of existing games simply employ shake recognition to trigger an event similar to a button press. As in other game consoles, the buttons of the Wii remote are heavily employed for triggering input events. Frequently, games use the Nunchuk attachment, which is designed to be held in the nondominant hand and adds more buttons, an analog joystick, and another accelerometer for independent motion sensing in each hand. In total, the Wii remote with Nunchuck attachment provides 13 digital inputs, 12 analog controls, and auditory, visual, and tactile feedback.

Communicating with the Wiimote

As mentioned in the previous section, communication between the controller and the computer is achieved using the HID standard. This enables one to get an enumeration of the reports that the controller understands using a HID descriptor block. Reports are conceptually equivalent to network ports assigned to specic services since they specify what kind of data is being transmitted and how it should be parsed. By specifying a report type and some data, requests can be sent to the Wiimote. Through requests, dierent states like reporting mode can be set. Also Wiimote features like LEDs and rumble can be enabled or disabled. However, since communication goes both ways, there are also report types to specify the format of the data which the Wiimote returns. Table 1, based on [29, 30] lists all of the supported report types. It should be noted that data report 0x30 and 0x33 are written explicitly since these are relevant to our project.

Most of the functions are less relevant to our project. However, one relevant report is the one controlling the data reporting mode (0x12). Hereby it is possible to control how and what data is being sent to the host. The Wiimote has two dierent ways of transmitting information to the host. As default the controller only sends data to the host when a state has changed, like when a button is pressed. However, it can be set to send data continuous even if nothing has changed since the last report was sent. This is done with a 10 ms interval, which means that 100 samples are sent per second. This is also the upper bound for the non-continuous report mode. The data reporting mode also controls what data is being sent from the controller. Several modes are available. The simplest mode 0x30 only sends button information while others send information regarding motion-sensing, expansion port and IR. As shown in Table 1, the report mode 0x33 sends button, motion-sensing and IR data. Since we are interested in receiving IR data as often as possible, this mode combined with continuous reporting is used.

Due to the way communication is done, there are a few things one should be aware of. When receiving packages from the Wiimote, they are stored in a buer. If they are read at the same rate as they are received, this buer can be ignored since it will have no inuence. However, if the packages are read less frequently, the size of the buer plays an important role. If the buer size is low, e.g. 1, then all but the most recent package at each sample point will be dropped and hence never read. This causes loss of data. However, a small buer size ensures that little or no delay is introduced since it is always the most recent package that is read. On the other hand if the buer size is large, the situation is quite opposite. Little or no data is lost, but large delays might occur. Since delays would make it hard to navigate precisely in a 3D world, it must be avoided in our application. Therefore we use a small buer of size 1. This may result in a few lost packages at times, but it is not critical in our application, since all packages contains the absolute position of the IR points as discussed in section 5.2.2. Another thing that should be noted is that the status of the rumble engine is send as the least signicant bit in every request report. Therefore it is important to make sure that this bit is 0 at all times since we never use the rumble feature.

Developing Custom Applications

Although Nintendo offers a relatively inexpensive development kit for the Wii console, its legal agreement severely limits the types of applications youre permitted to develop using its tools. Alternatively, you can quite easily connect the Wii remote to a personal computer via Bluetooth and immediately begin developing custom applications. The remotes compatibility with the Bluetooth HID specification manifests on the host computer as a joystick. Software libraries for connecting to a Wii remote, parsing the input report data, and configuring the controller are available for nearly every major development platform on Windows, Mac OS, and Linux. The open development community has created these libraries, and you can download them for free. Because these software APIs are in active development and might change rapidly, I wont discuss them in detail. Visit http://wiili.org and http://wiibrew.org for more information. Accessing the data is usually as simple as reading values from an array or an appropriately named member variable of a Wii remote class object, such as accelX = remote.accelerometer.x. The computer receives input reports 100 times per second, providing low-latency data. As the software libraries evolve, they might support event queues, derivative values, and utilities that compute useful transformation matrices or recognize gestures, thereby simplifying application development. In many cases, the most difficult part of this process is getting the Bluetooth pairing to occur successfully. Because the Wii remote isnt 100 percent HID compliant, it might work only with certain Bluetooth chipsets and driver software. However, once a pairing is successful, the configuration is typically quite reliable. After youve connected the Wii remote and installed the software library, developing custom applications is straightforward. The projects I describe in this article are C# Windows software applications using Brian Peeks Managed Library for the Wiimote.

Interpretation of DataWhen the position of the points is known, the users position can be calculated. The distance between the two LEDs is inverse proportional to the distance between the user and the screen. However, instead of just using the distance directly, Lee takes the size of the monitor into consideration. This is done by allowing the user to specify the size of her monitor. If the information is not given, a default value will be used.As can be seen in Figure 6 the distance from the user to the screen is inverse proportional to the tangent to half the viewing angle measured between the LEDs. To calculate this distance, the distance between the two LEDs is scaled with the size of the monitor and used to derive the relative distance between the user and the monitor. This distance is then used to calculate the position of the user relative to the monitor: By using the sine-function on the angle which indicates the distance between the LEDs on the x-axis, the x-coordinate relative to the camera-space is known. This method assumes, that the user is centred on the x-axis, when directly in front of the Wiimote. The y-coordinate in camera-space is calculated much the same way, but this coordinate is not assumed to be centred directly in front of the Wiimote. To take this into consideration an oset is used. This oset can also be changed by the user to t the setup used. In the above calculations we assume the position is centred around the origin. Since the Wiimote returns values ranging from 0 to 1023 and 0 to 767 for the x and y respectively, we must deduct 512 from the x and 384 from y.

< !!!! insert fig 6!!!!>>>

Combined, the x- and y-coordinate and distance from user to screen can give the camera position, camera target and camera up vector. From these three vectors the three axis of the coordinate system to be used can be determined:

< !!!! insert equations!!!!>>>

The view-matrix can now be determined from the axis by the following matrix:

< !!!! insert matrix!!!!>>>

The boundaries of the screen are also needed. The x- and y-coordinate gives the oset of the boundaries, while the distance gives the scale. If the oset was not used, objects closest to the user would move around, which must not happen. By using the oset it is ensured, that when the camera moves from side to side and up and down the front plane in the view-frustum remains the same. To keep the view-frustum the same when the camera is moved back and forth the boundaries must also be scaled with the front clipping plane. The boundaries are used to calculate the projection-matrix, which is constructed as follows:

< !!!! insert matrix!!!!>>>

It is noted, that the frustum given by this matrix is not necessarily the same to the left and right of the camera. However, this is needed because the plane closest to the user has to be locked to the monitor to achieve the eect of looking through a window. Calculating these two matrices nishes the interpretation of the data, since the world matrix needs not be modied.

The visualization is not particularly interesting. After the matrices needed for the transformation are calculated, the graphics are made as in most other 3D applications. This means that objects are shown by transforming them from the 3D space into 2D space by using the mentioned matrices. After this, they are rendered to the monitor.

Chapter 2Review of Literature

Paper 1: Experience in the Design and Development of a Game Based on Head-Tracking Input"Authors: Jeffrey Yim, Eric Qiu, T.C. Nicholas GrahamPublished : IACSS 2008. Calgary 3rd November, 2008.

This Paper shows that Tracking technologies, such as eye and head-tracking, provide novel techniques for interacting with video games. For instance, players can shoot with their eyes in a first person shooter using gaze-based input. Head-tracking systems allow players to look around a virtual cockpit by simply moving their head. However, tracking systems are typically based on expensive specialized equipment. The prohibitive costs of such systems have motivated the creation of low-cost head-tracking solutions using simple web cameras and infrared light detection. In this paper, we describe our experience developing a simple shooting game which incorporates such low-cost head-tracking technology. There have in recent years been significant advances in novel techniques for interacting with video games. One interesting direction involves the creation of more immersive ways of providing input to games, where players natural movements translate into in-game actions. Perhaps the most well-known of these are gesture-based interactions using a Wii Remote and movement-based interaction using a dance pad or Wii Balance Board. A very different style of approach has been the use of passive input devices that capture players focus of attention, and use this as a direct source of input to games. An example is eye-gaze control of video games, where for example, a player of a first person shooter can aim his gun simply by looking at the desired target. In this approach, players do not control the game through a physical input device; they simply look. Gaze-based input requires expensive equipment (e.g., a $28,000 Tobii eye tracker), and therefore cheaper approaches have been developed, such as head-tracking. Unlike eye trackers, head tracking systems cannot directly determine eye-gaze. Instead, player attention is approximated by capturing head position and orientation. In this manner, players can freely look around their cockpit in a flight simulator game. Head trackers, however, are not just low-fidelity substitutes for eye trackers. For instance, the act of dodging projectiles in a shooting game is naturally captured by lateral head movements.

Paper 2: Head tracking using a WiimoteAuthors: Kevin Hejn, Jens Peter RosenkvistPublished : COGAIN 28th March, 2008.

This paper presents an adaption of an approach to achieving a virtual reality like experience on a typical monitor using the Nintendo Wiimote. By using the infrared camera on the Wiimote combined with infrared light emitting diodes positioned on the users head, we achieve the effect that the content visible on the monitor adapts to the position of the head relative to the monitor. Thereby we achieve the same effect as when looking through a window; what is visible depends on the angle and distance from the window. In line with the growth of the game industry, the demands for realism in modern computer games increase as well. Realism is often raised through improved graphics, artificial intelligence, sound effects and similar. The player interaction devices are however, mostly limited to the use of mouse, keyboard and conventional game controllers. The possibilities have improved somewhat by the introduction of Wii1, NintendosR newest game console which was introduced in 2006. Now players can interact more directly and naturally with the games, thus achieving higher realism. This is done through the Wiimote controller which, among other things, features motion-sensing. However, critical areas are still left behind. There is for example poor access to equipment allowing the player to change the view perspective by moving her head. This can indeed already be achieved using available virtual reality (VR) equipment, but for most users it is not within an acceptable price range. It turns out that equipment allowing this kind of interaction does not necessarily have to be expensive. It can be achieved using an infrared and some light emitting diodes (LEDs). By placing the camera by the monitor and the LEDs by the head, the placement of the LEDs can be found, hereby allowing us to determine the users movement and changing the field of view according to this. This yields the illusion that the monitor is like a physical window into another room instead of just a static photo. This can increase the realism for especially 3D games dramatically. Normally registration of images is a difficult job and forms an entire research field, but when taking advantage of the build in IR camera and image processor of the Wiimote, it becomes quite simple to detect and track numerous positions. Furthermore the Bluetooth support and low price of the controller makes it easily accessible. Based on these arguments, we feel that the idea presented by Johnny Lee calls for further investigation. In thispaper we will therefore try to mimic his work and determine the field of viewin a 3D game like world from the available data. Through a user study we furthermore wish to evaluate whether our solution is suited for interaction in a 3D world.The purpose of this paper was to mimic the work of Johnny Lee and implement a solution for performing headtracking using the Wiimote. The goal of the headtracking was to allow users to interact with a 3D world and give much of the same feeling as when looking through a window. this paper described the Wiimote and the how communication with it is performed as well as how the returned data can be used to navigate in a 3D world. And also made an analysis of the problem as well as of the solution of Johnny Lee, discussed his approach and aspects that could be improved; namely robustness and noise reduction. Based on the analysis, described the solution and made suggestions to improve the robustness andnoise problems.

Chapter 3Proposed Methodology

In this section we will first give a general analysis of the problem we are trying to solve. This is followed by a analysis of the solution of Johnny Lee and the methods used. Finally this will result in a description of the solution we will use to achieve our goal.

Figure 3.1: Schematic block diagram of apparatus.

Figure 3.2: Block diagram of Interfacing Wiimote console.

3.1 Defining the desired effect

As described briefly in the introduction, our goal is to use headtracking to achieve a VR like effect when looking at a monitor. The effect we wish to obtain can be clarified with the following analogy which is also used in . A traditional monitor can be compared to a photo in a frame. No matter at what distance or angle you watch the motive, you see exactly the same content. This is due to the 2D nature of the photo or image displayed, even though the original content was three dimensional. However, if you remove the photo from the frame, this changes. Now, what you see through the frame depends on the angle and distance that the frame is viewed, exactly like when looking through a window. This effect is what we wish to achieve on the monitor. Depending on the position of the user, the visible content changes to reflect her movement.

3.1.1 Movement scenarios

The movement that the user can make can be separated into two general categories; changes in the angle between the front of the monitor and the user and changes in the distance between the monitor and the user. The change of angle is an effect of the user moving around in a constant dis- tance from the centre of the screen (i.e. the position of the Wiimote). When this occurs, some elements should vanish while others should become visible. Again, using a window as an example; if you stand in front of a window and move to the left, you can see more to the right on the other side of thewindow and vice versa. The situation is illustrated from above in Figure 4(a).

Figure 3.3: The change in the visible field when the user moves to a different angle relative to (a) a window and (b) a monitor.

As a contrast, the normal scenario when using a monitor is shown in Figure 4(b) where it is obvious that the angle of view has no influence and thus seems unrealistic. The same principles also apply when lowering or raising your head; you can see more of the sky through a window when crouching than when standing. The other category, change of distance, is a different effect as the distance between the monitor and the user determines how much of an image should be visible overall. Returning to the window example: the closer one is to a window, the more one can see outside the window in all directions. The situation is illustrated in Figure 3.4.

Figure 3.4: The change in the visible field when the user moves closer to (a) a window and(b) a monitor.

As one moves closer to the window, more of the outside becomes visible. As before, Figure is provided as a contrast, showing how distance has no effect when using a traditional monitor.

To enable the kind of interaction described above, the computer needs to know the position of the users head relative to the monitor. This problem is known as headtracking . As mentioned, headtracking in general is difficult since the process of finding areas of interest and registering several images is complicated. However due to the IR camera and onboard image processor of the Wiimote, the task is significantly simplified. As described in section the data returned from the Wiimote contains absolute coordinates of up to four points and optionally size estimates. Since little information can be obtained when only one point is used, as the size estimate is too coarse, two points are required. This is also the amount of points recognized when using the sensor bar. This approach is the foundation in the solution of Johnny Lee as well as in our solution.

3.1.2 Interpretation of data

When the position of the points is known, the users position can be calculated. The distance between the two LEDs is inverse proportional to the distance between the user and the screen. However, instead of just using the distance directly, Lee takes the size of the monitor into consideration. This is done by allowing the user to specify the size of her monitor. If the information is not given, a default value will be used. As can be seen in Figure 6 the distance from the user to the screen is inverse proportional to the tangent to half the viewing angle measured between the LEDs. To calculate this distance, the distance between the two LEDs is scaled with the size of the monitor and used to derive the relative distance between the user and the monitor. This distance is then used to calculate the position of the user relative to the monitor: By using the sine-function on the angle which indicates the distance between the LEDs on the x-axis, the x-coordinate relative to the camera-space is known. This method assumes, that the user is centred on the x-axis, when directly in front of the Wiimote. The y-coordinate in camera-space is calculated much the same way, but this coordinate is not assumed to be centred directly in front of the Wiimote. To take this into consideration an offset is used. This offset can also be changed by the user to fit the setup used. In the above calculations we assume the position is centred around the origin.

Figure 3.4 Interpretting the LEDs position.

Combined, the x- and y-coordinate and distance from user to screen can give the camera position, camera target and camera up vector. From these three vectors the three axis of the coordinate system to be used can be determined:

This means that objects are shown by transforming them from the 3D space into 2D space by using the mentioned matrices. After this, they are rendered to the monitor.

Chapter _Conclusions

At rest, the Wii Remotes accelerometer detects gravitys pull and can thus be used as an inclinometer. It provides a good estimate of the pitch and roll (together referred to here as inclination) of the Remote leaving only yaw (azimuth) unconstrained. Using this constraint, we develop a technique to register an object of known geometry to an image with only three or four points. This technique is then applied to create a self-calibrating six-degree-of-freedom input device.

Instead of attempting to intelligently determine correspondences, the small number of unknowns allows for a guess-and-check approach. Each possible mapping of markers on the tracked object (which we call an artifact) to observed points is attempted and the resulting transformation is checked for consistency with the observed points and inclinations.

The purpose of this project is to implement a solution for performing headtracking using the Wiimote. The goal of the headtracking was to allow users to interact with a 3D world and give much of the same feeling as when looking through a window.

AppendixDetailed information, lengthy derivations, raw experimental observations etc. are to be presented in the separate appendices, which shall be numbered in Roman Capitals (e.g. Appendix I).

Chapter _References[1] Jeffrey Yim, Eric Qiu, T.C. Nicholas Graham Experience in the Design and Development of a Game Based on Head-Tracking Input" IACSS 2008. Calgary 3rd November, 2008.

[2] Kevin Hejn, Jens Peter Rosenkvist Head tracking using a Wiimote COGAIN 28th March, 2008.

[3] Sreeram Sreedharan, Edmund S. Zurita, and Beryl Plimmer. 3D in- put for 3D worlds. In OZCHI 07: Proceedings of the 2007 conference of the computer-human interaction special interest group (CHISIG) of Australia on Computer-human interaction: design: activities, artifacts and environments, pages 227230. ACM, 2007.

[4] Akihiko Shirai, Erik Geslin, and Simon Richir. Wiimedia: motion anal- ysis methods and applications using a consumer video game controller. In Sandbox 07: Proceedings of the 2007 ACM SIGGRAPH symposium on Video games, pages 133140. ACM, 2007.

[5]R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision, Cambridge Univ. Press, 2003.

[6]R. Raskar, G. Welch, and K.-L. Low,Shader Lamps: Animating Real Objects with Image-Based Illumination, Proc. Eurographics Workshop on Rendering, Springer, 2001, pp. 89102.

[7] James D. Foley, Andries van Dam, Steven K. Feiner, and John F.Hughes. Computer Graphics - Principles And Practice. Addison Wesley, 2nd edition edition, 1996. [8] Ralph L. Rosnow and Robert Rosenthal. Beginning Behavioral Research- A Conceptual Primer. Prentice Hall, 5th edition edition, 2005.

AcknowledgementsI am profoundly grateful to Prof. Prof. Rajan Deshmukh for his expert guidance and continuous encouragement throughout to see that this project rights its target.

I would like to express deepest appreciation towards Dr. Varsha Shah, Principal RCOE, Mumbai and Prof. Dinesh Deore, Head of the Computer Engineering Department whose invaluable guidance supported me in this project.

At last I must express my sincere heartfelt gratitude to all the staff members of Computer Engineering Department who helped us directly or indirectly during this course of work.

Adil Khan (66)Huzefa Saifee (42)Roshan Shetty (49)