Smart Objects for Human Computer Interaction,

Experimental Study

Doggen, Jeroen; Neefs, Jef; Brands, Enzo; Peeters, Tom;Bracke, Jerry; Smets, Marc; Van der Schueren, Filip

January 31, 2012

Artesis University College of Antwerp, Department of Applied Engineeringand TechnologyPaardenmarkt 92, B–2000 ANTWERPENJeroen.Doggen@artesis.be

AbstractThe fields of embedded systems and networking are being merged,giving rise to the “Internet Of Things”, built with smart objects thatallow people to interact with everyday objects and vice-versa. Wedeveloped a cube, equipped with an Arduino based development board,various sensors and an XBee wireless interface, which enables us todevelop new methods of human computer interaction. The cube wastested as an input controller for several computer games. The softwarewas released under an open source license. This system will be usedto develop new sensor driven applications.Keywords: Smart Objects, Human Computer Interaction, MotionSensing, Distance Sensing, Arduino.

1 Introduction

Advances in technology development, made possible by combining the fieldsof embedded systems, networking and sensor systems, give rise to new typesof applications. Wireless networks of sensor equipped embedded systems,often called smart objects, are connected together to form the “Internet OfThings”.Household appliances and other consumer products have been equipped withelectronics for decades. These electronics contain a lot of useful informationabout the objects they are embedded in. Most of this information is relayedlocally to the users via displays and control lights who then can respond tothat information accordingly.Radio-frequency identification (RFID) tags were the first mainstream appli-cation of electronic tagging of everyday non-intelligent objects. RFID tagsprovide static and rigid information about clothing and other supermarket orwarehouse goods in general. These tags typically contain a limited amount

of information: price, object identification and manufacturing date. In themeanwhile we can find RFID tags with up to 1 MB (active) and 8 kb(passive) of storage space. Near Field Communication technology (NFC)extended the use of RFID by improving and standardising the interactionamongst reader and tag and readers themselves allowing more complexapplications [1].The field of Micro-Electrical-Mechanical Systems (MEMS), where very smallmechanical devices allow on chip measurement of motion and rotation ofrigid objects, gave rise to straightforward motion related applications, e.g.collision detection [2].The next logical step in technology development is mixing the fields ofinformation sharing by passive tags and information gathering by activedevices. Unlike the data in most of the contemporary RFID tag applications,the information contained in smart objects is very diverse and changesrapidly. These smart objects will not only know what they are, but alsowhere they are [3]; they will sense their environment and communicate withother smart objects to increase sensing efficiency [4] and will interact in newways with the end-user [5].

2 Problem Statement

Our primary goal is to combine several sensor technologies and data sourcesto test new methods of human computer interaction.In a previous project, we implemented a customised implementation of afrustrated total internal reflection based multi-touch table [6].The first idea to improve the functionality of this multi-touch table, was toattach visual masks to all objects. The system could identify the objects byprocessing the video stream and recognising the masks.The second idea was to make an intelligent object that could deliver theadditional context information: the acceleration, rotation and position ofthe object.By combining these concepts with wireless sensor network technology andby performing on-device sensor preprocessing, we can design new types ofapplications.The drawback of the visual mask implementation is that the multi-touchtable needed a projector and a camera for the mask detection. This makesthe implementation bulky, expensive and complex to build. On top of thatall objects need to have a visible mask on one or more sides so they canbe detected by the camera and software. Recently backlit Infrared (IR)touchscreens removed the need for a camera. But this new technology isstill quite expensive, a table sized model would cost you as much as threehigh-end liquid crystal display (LCD) televisions of the same size.We believe that the smart object solution, implementing sensing and

processing technology can be less expensive and is able to increase thepossibilities of the setup. Figure 1 shows a mockup of our envisionedimplementation.A smart object, equipped with a triple axis accelerometer, is able to detectwhich side of the object is facing upwards, implementing a gyroscope onthis smart object makes the object “orientation aware”. Well placed distancesensors allow the object to detect its height above the screen when picked up.This removes the need for a projector and camera or an expensive IR backlitdisplay, because no masks need to be detected to determine the object’sorientation and movement. The smart object could have features that werepreviously impossible: e.g. activating different functions depending on theupward facing side. Extra inputs can be derived from the device withoutthe device having to touch the screen, e.g.: tap, pick-up, shake and drop.The possibilities are only limited by the sensors one can attach to such asmart object.

Figure 1: Smart Cube for Human Computer Interaction

3 System Design

For this experimental study we developed a smart object based on the pop-ular Arduino development platform, which we equipped with an accelerom-eter, a gyroscope and distance sensors. This platform has gained increasedpopularity over the last years. The Arduino open-source community hasover 64000 registered users and an abundance of user submitted libraries.

3.1 System Architecture

The complete system consists of two main blocks: a wireless module / cubeand a PC equipped with a XBee wireless interface.The movements of the cube are detected using motion sensors. We imple-mented two types of motion sensors: a triple axis accelerometer and a triple

Figure 2: System Architecture

axis gyroscope. The basic design uses an analog accelerometer, the signalsfrom the three axes are sampled using three of the eight channels of the 10-bit analog-to-digital converter (ADC) on the ATmega328P microcontroller[7]. The gyroscope on the other hand is a digital one, which sends its dataover the Inter-Integrated Circuit interface interface (I2C) to the developmentboard. The microcontroller runs the embedded code to process sensor anduser events, making the object “smart”. The microcontroller is connectedto the XBee module over the serial interface. Since the 802.15.4 interface isnot commonly available on computers, we need an extension on this sideto connect our smart object to the computer. We connected an XBeemodule to an XBee shield, which in turn is connected to an Arduino boardwhere the microcontroller has been removed. This module provides a serialport to the computer over the Universal Serial Bus (USB) interface usinga Future Technology Devices International (FTDI) chip. Now the 802.15.4interface is available on the computer. On that computer we then run aserver application whose task is to forward the serial data towards clientapplications that make a connection over a Transmission Control Protocol(TCP) socket. The library to connect to the system and several demoapplications are described in section 3.3.2 and section 4.3.

3.2 Hardware Specifications

A typical smart object contains five building blocks: CPU, sensors, wirelesscommunication interface, memory and a power source. In this project wechose these blocks considering the following constraints: price, ease of use,flexibility and energy consumption.

Figure 3: Partially Disassembled Cube: Sensors, XBee and Microcontroller Board

Seeeduino Development Board The price tag has to be low, around$100, to allow production of custom printed circuit board (PCB) variantson the design. The schematics will be available online under an opensource license. Since these designs will also be constructed and altered byundergraduate students, we needed a well documented microcontroller withan easy to use yet flexible and powerful development environment. Hence wechose an Arduino compatible development board: Seeeduino. Seeeduino’sdesign is based on the original Arduino, existing software and hardware is100% compatible, but on the hardware side, it has many improvements, e.g.power efficient surface mount device (SMD) components and extra analogand digital I/O pins. Its most important hardware specifications are: 14digital I/O pins, 8 analog inputs and a 16 MHz crystal oscillator [8].

XBee Wireless Interface: The system was not designed to be ultra-low-power, but it should be able to operate over a period of several daysusing 2 AA batteries. Hence we chose the MaxStream/Digi XBee low-powerwireless communication modules to connect the cube to the basestation [9].

Analog Accelerometer The Freescale Semiconductor “MMA7361” ac-celerometer is a cheap, triple axis sensor that can be configured for ±1.5 gand ± 6 g operation allowing a broad range of applications [10].

Digital Gyroscope The InvenSense “ITG-3200” gyroscope is a single-chip, digital output, triple axis MEMS sensor, optimised for humancomputer interaction. Internally, the chip uses a three channel 16-bit ADC.

The chip connects to the outside world over a Fast-Mode I2C interface,running at 400 kb/s [11].

Infrared Distance Sensor The Sharp GP2Y0A21YK infrared Distancesensor sends out a pulse of IR light that travels out in the field of view andhits an object. The reflected light creates a triangle between the point ofreflection, the emitter, and the detector. The angles in this triangle varybased on the distance to the object. A precision lens directs the reflectedlight onto various portions of the enclosed linear CCD array based on theangle. The sensor varies its output voltage accordingly [12].

3.3 Software Libraries

The Arduino library provides high level events or sensor data based on themovement of a smart object. The XNA library provides easy access to theseevents from a C# software development perspective.

3.3.1 Arduino Libraries

The Arduino development environment makes it straightforward to startembedded development, create libraries and share them with the onlinecommunity. We used an existing software library to interface with the ITG-3200 Gyro [13]. Because there was no library available for the MMA7361accelerometer, we developed our own library and published it on theArduino Forum [14]. After 9 months we had around 600 downloads of ouracceleroMMA7361 library.The distance sensor library provides a unified and easy to use interface forultrasonic and infrared distance sensors.The libraries are published as open source projects on Google code [15] [16].

AcceleroMMA7361: Library for retrieving data from the MMA7361accelerometer. It provides sensor data as a plain binary ADC output, asan absolute output voltage and in reference to earth’s gravitation. It alsodelivers a calibrate function to determine a horizontal reference position, bysetting static offsets on the accelerometer axes in case of a slightly tiltedstarting position at boot time.

AcceleroDice: Library which determines the upward facing side ofthe cube. The code sample in listing 1 shows a minimal application,implemented using this library. In the “setup phase”, the cube is initialisedand calibrated using the AcceleroMMA7361 library. During operation, thesystem detects which axes the gravitational acceleration is working on andcalculates its upward facing side based on averaging and thresholding. Thisinformation is updated on each movement.

Listing 1: AcceleroDice: Minimal Implementation

1 #include ”AcceleroMMA7361 . h”2 #include ”Acce le roDice . h”3 #include <Arduino . h>45 AcceleroMMA7361 a c c e l e r o ;6 Acce le roDice d i c e ;78 void setup ( )9 {

10 S e r i a l . begin ( 9600 ) ;11 a c c e l e r o . begin (13 , 12 , 11 , 10 , A0 , A1 , A2 ) ;12 a c c e l e r o . setARefVoltage ( 3 . 3 ) ; //AREF vo l t a g e = 3.3V13 a c c e l e r o . s e t S e n s i t i v i t y (LOW) ; // S e n s i t i v i t y = +/−6G14 a c c e l e r o . c a l i b r a t e ( ) ;15 d i c e . begin ( a c c e l e r o ) ;16 }1718 void loop ( )19 {20 i f ( d i c e . sideChanged ( ) )21 {22 S e r i a l . p r i n t l n ( d i c e . ge tS ide ( ) ) ;23 }24 }

AcceleroArrow: Library for using the accelerometer as the arrow key-board input. The library detects sudden movement and sends an “Arrow-Pushed” event when a certain acceleration threshold is reached. It stopssending the event when a total deceleration, resulting in the opposite of thefirst acceleration, is reached.

Distance Sensor Library: The analog sensor’s output voltage is used toget the corresponding distances from a lookup table. The first version ofthe lookup table was derived from an approximation of the transfer functionbased on the sensor datasheet. This approach resulted in inaccurate results,possibly due to inter-sensor differences.To improve our results, we tested several sensors by reading their outputvoltages while placing an object at known distances ranging from 5 cm upto 80 cm. The summarised results in figure 4 show the average sensor outputand two polynomial approximations for the transfer functions: one for thesteep part and one for the flat part of the slope.We calculated the remaining points of the 256 byte lookup table using thetwo transfer functions. This static information is placed in code memorytaking the limited amount of random access memory on the ATmega328Pinto account.

Figure 4: Transfer Functions for the GP2Y0A21YK Infrared Distance Sensor

3.3.2 XNA Library

Microsoft XNA is a set of software tools that facilitate video gamedevelopment and management. XNA attempts to free game developers fromwriting low-level code.Our XNA library uses a TCP/IP socket connection to interface with thesmart objects. This gives software developers easy access the features of themodule, without extended knowledge of the system.It can be used to read the X, Y and Z values, to detect a jump and togenerate “arrow events”.

4 Applications

We developed several tools and performed experiments to calibrate thesensors and verify proper system operation while developing the Arduinolibraries. The libraries were used to modify 3 in-house developed computergames: a 3-D flight simulator, a 2-D platform game and a Pac-man clone.

4.1 Signal Conditioning

Most MEMS inertial sensors are comb based mechanical structures fab-ricated on silicon wafers. The complex fabrication steps required duringMEMS production often result in fabrication deficiencies causing signaldifferences between individual sensors. These sensor errors have beenstudied thoroughly [17] and several possible solutions have been proposed[18]. Expensive sensors are often factory calibrated to compensate for theseundesirable effects.We implemented a rudimentary software calibration method in the ac-celeroMMA7361 Arduino library to take individual sensor differences andmechanical misalignment into account.Based on user-feedback, we decided to implement and compare three simplesignal processing algorithms: variable-length weighted moving average filter,

Bessel and Chebychev low-pass filters.At first these filters were implemented and optimised in a C# application. Ina later version the filters were moved to the microcontroller. As illustratedin figure 5 these simple filters don’t have the desired effect when used tofilter the accelerometer data. The low-pass filter does not change the signalsignificantly and the averaging filter introduces to much delay.The next steps in development will involve on-device Kalman filtering andallowing the smart object to adjust signal processing settings based on theapplication requirements.The library is published as an open source project on Google code [19].

Figure 5: Original data - low-pass filter - moving average filter

4.2 Software Tools

SerialMonitor: This monitoring tool was developed to log and analysesensor data and serial communication. The full system does not need thistool, but it was used extensively during development.

SerialForwarder: This application, developed using the processing pro-gramming language [20], runs on the XBee-enabled computer. It connectsto the serial port and provides two way communication between a TCP/IPsocket and the serial port. This allows most network enabled applicationsto communicate with our cube.

4.3 Demo Applications

In the flight simulator, we control the movement of an airplane, flying overa 3-D landscape and passing though rings using the sensor data as a joystickas shown in figure 6.We developed the AcceleroArrow Arduino library to control the Pac-manclone.The 2-D platform game required a new feature on the smart object: jumpdetection. This was implemented by detecting sudden upward movements,with a direction opposed to earth’s gravitational pull, independent of theobject’s orientation.

Figure 6: Space ship game

5 Future Work

The next steps in development will be enhancing existing non-smart systemswith our cube. Two applications that are currently being developed are asensor enhanced radio-controlled car and a model train railroad where trainscommunicate to achieve common goals. Motion sensors are used for collisiondetection and IR sensors for collision avoidance.To test the reliability and performance of the measuring system, we needto generate movement data under the same mechanical conditions, withinan acceptable degree of error. For this goal we plan to put the cube ona turntable with controllable rotation speed, in this way the sensors willcomplete a periodical motion in a two-axis system. By controlling andmeasuring the speed of the turntable we can calculate a reference valuefor the data we can expect from the motion sensors. From this test setup weplan to evaluate error rates for multiple speeds and processing algorithms.The distance sensor library needs some extra fine-tuning to optimise thelibrary for code size. We plan to replace the 256 byte lookup table witha smaller table by performing linear interpolation between a limited setof points. This might be feasible because of the nearly linear relationshipbetween the ADC output and the reciprocal of the distance.This will allow us to perform very rudimentary run-time calibration andthereby automatically supporting various IR distance sensors.Currently, most of the features of the smart objects are static and determinedat compilation time. We plan to develop a more extensive set of commandsto allow the end-user applications to adjust the on-device processing, e.g.:adjust filter parameters or more complex system behavior.On the hardware side we plan to reduce the power consumption by replacingthe XBee module with NRF24L01+ 2.4 GHz low power transceiver producedby Nordic Semiconductor [21].

We plan to extend the sensor library enabling other developers who lackthe fundamental signal processing skills to use digital filters in theirapplications.

6 Conclusion

We developed a cube, a smart object, based on the popular Arduinohardware platform. This smart cube, equipped with motion sensors and amicrocontroller, processes its sensor inputs and provides movement data asrequested by an end-user application. The end-user application is connectedover a TCP/IP network to a control server, which in turn is connected tothe cube using a wireless ZigBee network connection.We developed software libraries for the MMA7361 accelerometer and theSRF04 and GP2Y0A21YK distance sensors. The source code of theselibraries was introduced on the Arduino Forum and is now publicly availableunder the GNU Lesser General Public License on the Google code website.The cube was tested as an input controller for several different computergames. New features where implemented in the software libraries, based onthe needs of these computer games.This system will be used to develop new sensor driven applications and toenhance existing non-smart objects to enable completely new features.From an educational perspective, this design offers a lot of opportunities forstudent projects. Extended versions will house multiple sensors, implementvarious signal processing methods and offer many other opportunities.

References

[1] Neefs, Jef; Schrooyen,Frederik; Doggen, Jeroen and Renckens, Karel;Paper ticketing vs. Electronic Ticketing based on off-line system“Tapango”, Second International IEEE Workshop on Near FieldCommunication, pp. 3-8, 2010.

[2] Jaw-Kuen Shiau; Wei-Yan Luo and Long-Kuang Lee, “Three-Axis Electronic Collision Sensor for Airborn Application”, 2ndInternational IEEE Symposium on Systems and Control in Aerospaceand Astronautics (ISSCAA), pp. 1-6, 2008.

[3] Davidson, Pavel; Collin, Jussi and Takala, Jarmo; “Application ofparticle filters for indoor positioning using floor plans”, UbiquitousPositioning Indoor Navigation and Location Based Service (UPINLBS),pp 1-4, 2010.

[4] Fourati, H.; Manamanni, N.; Afilal, L.; Handrich, Y.; “A NonlinearFiltering Approach for the Attitude and Dynamic Body AccelerationEstimation Based on Inertial and Magnetic Sensors: Bio-LoggingApplication”, IEEE Sensors Journal, Vol. 11, pp. 233-244, 2011.

[5] Siciliano, P.; Leone, A.; Diraco, G.; Distante, C.; Malfatti, M.; Gonzo,L.; Grassi, M.; Lombardi, A.; Rescio, G.; and Malcovati, P.; “Anetworked multisensor system for ambient assisted living application”;3rd International IEEE Workshop on Advances in sensors and Interfaces(IWASI), pp. 139-143, 2009.

[6] Han, Jefferson Y.; “Low Cost Multi-Touch Sensing through FrustratedTotal Internal Reflection.”; 18th annual ACM symposium on Userinterface software and technology (UIST ’05), pp. 115-118, 2005.

[7] 8-bit Atmel 328P Microcontroller, Atmel Corporation.[8] Seeeduino, Arduino compatible development board; http://www.

seeedstudio.com.[9] XBee/XBee-PRO OEM RF Modules, www.digi.com/pdf/ds_

xbeemultipointmodules.pdf.[10] MMA7361: 3-Axis Acceleration Sensor; Freescale Semiconductor;

http://www.freescale.com

[11] ITG-3200 Integrated Triple-Axis Digital-Output Gyroscope; In-venSense; http://invensense.com

[12] GP2Y0A21YK Distance Sensor, Sharp Electronics; http:

//sharp-world.com/

[13] Arduino driver for the ITG-3200 3 axis MEMS gyroscope byInvenSense; http://code.google.com/p/itg-3200driver.

[14] Neefs, Jef; Library for the Freescale Semiconductor MMA7361Accelerometer; http://www.arduino.cc/cgi-bin/yabb2/YaBB.pl?

num=1291717903/0.[15] Neefs, Jef; Doggen, Jeroen; Arduino library for the MMA7361

3-axis MEMS accelerometer; http://code.google.com/p/

mma7361-library.[16] Doggen, Jeroen; Arduino library for distance sensors; http://code.

google.com/p/arduino-distance-sensor-library/.[17] Park, M. and Gao, Y., “Error Analysis and Stochastic Modeling of Low-

cost MEMS Accelerometer,” The Journal of Intelligent and RoboticSystems, Vol. 46, pp. 27-41, 2006.

[18] S. C. Shen, C. J. Chen, H. J. Huang and C.T. Pan, “Evaluationof MEMS Inertial Sensor Module for Underwater Vehicle NavigationApplication”, International IEEE Conference on Mechanic Automationand Control Engineering (MACE), pp. 3807-3810, 2010.

[19] Doggen, Jeroen; Arduino library for signal filtering; http://code.

google.com/p/arduino-signal-filtering-library.[20] Processing Programming Language; http://processing.org.[21] nRF24L01+ Ultra low power 2.4GHz RF Transceiver; Nordic

Semiconductor; http://www.nordicsemi.com/