Bringing Virtual Events into Real Life in SecondLife Home Automation System

SK Alamgir Hossain, A S M Mahfujur Rahman, Abdulmotaleb El SaddikMultimedia Communications Research Laboratory

University of OttawaOttawa, Canada

{skahossain, kafi, abed}@mcrlab.uottawa.ca

Abstract—In this paper we propose the design and develop-ment of a prototype system to monitor and control a physicalhome and to bridge the interaction gap between the virtual andreal world device control mechanism. We created a virtual 3Dhome in Second Life that mimics the look of a physical home. Inthe physical home environment different devices and sensors areconnected in order to ensure a safe and automated home. Anyevent that occurs in the physical space of the smart home is thensynchronized with the virtual environment. More importantly, thevirtual home interface provides the option to control the physicalsmart devices. By using the Second Life virtual interface thehome owner have a better look to monitor or control the homeappliances. As per the initial experimentation, we found out thatthe proposed approach of Second Life based home automationsystem is appealing and useful to the user.

Index Terms—Smart Home, Second Life, 3D Virtual Environ-ment, Surveillance, Virtual Reality

I. INTRODUCTION

A 3D virtual world is a computer based simulated multi-media environment usually running over the Internet, wherea user can access and interact with the other users by usinga 3D virtual persona called avatar. Second Life1, which isdeveloped by Linden Lab, is one of the most popular 3Dvirtual environment that acts as a medium of social interaction.Here, people can build their virtual 3D home, customize it with3D furniture and populate that with 3D interactive devices.With the growing popularity of Second Life, users often designtheir virtual 3D homes mimicking that of their real homes.Our proposed method can be used to bridge the gap betweenthe real home and the respective 3D virtual home in SecondLife by incorporating services that control and synchronize thecommunication events to and from the smart devices. In thisapproach, through the interaction of the virtual home elementsthe user can intuitively maneuver the real world devices ina smart environment. In smart environment different wiredand wireless sensory devices such as temperature, humidity,pressure sensors etc. are installed. By using these sensory datait is possible to measure different environment conditions andmake decisions. The prices of these sensors and automationdevices such as X102, WiFi are decreasing with time. Atthe same time, their usage to provide control and variousentertainment facilities in smart spaces are becoming hugely

1Second Life, http://secondlife.com2X10, http://www.x10.com/homepage.htm

popular. A graphical user interface (GUI) which is a mediumto interact with programs plays an important role especiallyto control physical space elements like smart home. However,controlling a physical space by using the traditional icon basedGUI might not be sufficient for effective management of arange of smart physical devices. In order to efficiently controldifferent intelligent physical devices, researchers have placedattention on the intuitive GUI design and explored its usabilityissues [1] [2].

Collaboration between physical and virtual world is notnew. Fleck et. al [3] presented the first notable work forcontrolling the physical space monitoring by using a virtualuser interface. The authors presented an integrated 3D worldmodel, where the physical objects were collected throughinter-camera tracking system and embedded as live texturein the 3D world. Presumably the interface minimized theoverhead of the observing person who was serving the systemto understand the events that happened in the real environ-ment. Another notable work related to Augmented VirtualEnvironment which fused dynamic imagery of the real worldobject into the 3D models is [4] in which authors presented thetechnical challenges about moving object detection, trackingand 3D visualization for effective dynamic event comparison.Many previous works found the adoption of virtual realitytechnologies in educational practices often presented problemsthat addressed the difficulties in navigation [5] while using3D interfaces. Users risked to spend a lot of time to controltheir movements, to reach to a specific location or to gain therequired point of view.

User interface is not only important for efficient control ofthe system but also it depends on the users’ ages, intelligences.In the paper Bin Zhang et. al. [6] showed how those factorsaffect the design of user interface. Those factors are veryimportant and have a high impact in users’ cognitive taskinside a home. The importance of 3D user interface (e.g. 3dvirtual world) over a 2D interface were perfectly describedin [1], in which user shows that how a 3D user interfacecan improve intuitively and adaptivity of smart home controlinstead of ordinary 2D user interface.

Simulating real world environment in the 3D virtual worldsimilar to Second Life is also popular now. Recently Univer-sity of Arkansas installed a prototype system called VirtualRazorback [7] in Second Life which is a virtual hospital

978-1-61284-890-7/11/$26.00 ©2011 IEEE

Fig. 1. An overview of the proposed Second Life home automation architecture. Each physical device has a virtual 3D representation in the Second Lifewhere user can control the physical devices through the virtual objects as well as the physical devices can change the animation by using the home automationsservices.

for modeling certain aspects of health care and a hospital’ssupply chain, from receiving goods to patient care. The virtualhospital has virtual furniture and equipments like the realworld hospitals. In order to navigate and control the 3D virtualenvironment based interfaces user often requires training.However, as the Second Life users are already adapted, theusers have almost no learning with the interface. Moreover,a flexible authoring capabilities of 3D objects [8], [9], builtin security features, scalability and portability features makeSecond Life as the most popular 3D virtual environment.Currently it has 22 million active subscribers as of February2011 3. In this paper we propose the design and developmentof a flexible virtual smart home like [1] [2] for controlling thereal life devices through a usable 3D user interface.

Our contribution in this paper is two-fold. First, in order tobridge the gap between virtual and real, we present a SecondLife Client add-on, where we provide flexible architecture forcontrolling the physical space. Second, we incorporate virtualannotation mechanism for the Second Life object for physicalspace monitoring.

The rest of this paper is organized as: at first, in Section IIwe illustrate the various components of our proposed systemand provide a general overview of the system. Further in Sec-tion III we describe the implementation issues, developmentchallenges of different modules and some evaluation strategies.At the end we provide conclusion of the paper in Section IVand state some possible future work directions.

II. SECOND LIFE HOME AUTOMATION SYSTEM

In this section we present the Second Life based homeautomation system architecture and the different componentsof the system. At first in Section II-A, we illustrate thearchitecture of the proposed system. Further in Section II-B,we describe the object annotation mechanism in Second Life.

3SL online user count, http://secondlife.com/xmlhttp/secondlife.php

A. General Architecture

Our system comprises of three primary components that areSecond Life Viewer plug-in, home automation web servicesand the point-to-point physical connection between servicesto the wireless and/or wired devices. In order to communicatewith the core part of the second life client program calledviewer we developed a Second Life interaction event handleras an add-on to the viewer. This coupling architecture providesthe option to listen to the communication channel of theSecond Life without affecting the functionality of the coresystem. Furthermore, the event controller interacts with theother parts of the home automation or surveillance systemby using Simple Object Access Protocol (SOAP). Interactionevent handler receives and/or transmits messages to/frommessage transmission module and captures the events that aregenerated from the message transmission module.

Fig. 2. The flexible object annotation scheme allows the user to annotateobject and allow setting animation properties. When interacted by home usersor automated triggering, the users will see the rendered animation on thescreen.

The event handler performs actions by using text basedmessaging protocol. The Message Transmission Module cap-tures all the messages that are generated in the core section

of the Second Life. When through the avatar the user issuesevents in the 3D environment, e.g., a click event to open adoor, the message transmission module captures those eventsand transfers the event messages to the nearby interactionevent handler for processing. The event handler module de-termines the particular event handling routine for a specificevent and then packs the event-handling message with thehandling routine. The handler then sends the packet to theinteraction event decoder to communicate with the actualhome automation services. Message transmission module alsoreceives animation data from the animation parcel managerand generates animation sequence for the objects in the 3Dvirtual world. For example, when a person opens a physicaldoor by using the RFID reader then a synchronized door openanimation is rendered in the virtual environment. Similarly,when a user clicks a door in the virtual environment it opensthe physical door of the real home.

Fig. 3. An overview of the target object specific interaction rules stored (andcould be shared) in an XML file.

B. Home Annotation in Second Life

In order to send or receive synchronized communicationto/from the real or the Second Life viewer we create 3D virtualobjects in Second Life for each of the physical devices thatwant to control. Further, we define annotation rules into these3D objects and later on attach the annotated files to the object.The techniques are described further in the following sectionsin details.

1) Object Annotation: In our system we annotate SecondLife 3D objects and specify the corresponding physical deviceaddresses. Every object in the real life has a unique id bywhich smart home services can control it. Physical devicesare connected with the home services by using X10 or WiFiconnections. The 3D objects are annotated using the Second

Life’s built-in script annotation mechanism that is demon-strated in Figure 2, where the 3D lamp is annotated withthe help of a separate annotation file. The annotation file isan XML file that specifies the animation unique identifier(UUID), the animation file (BVH), the speed of animation,duration of animation etc. This file also specifies the physicaldevice specific data, e.g., device address, name, type etc. TheBVH or Biovision Hierarchical is a file format that containsmotion capture data. The BVH format, which is developedby BioVision Inc, is actively used in computer animationapplications.

Fig. 4. Multi-region annotation in a 3D fan and specified a physical lampaddress which is connected to a X10 or Wi-Fi.

Figure 3 depicting a sample XML specification that demon-strates a door-open-close scenario where two animations”doorOpen1” and ”doorClose1” animations are specified foropen/close the real door that is connected with a X10 appliancemodule. Here the device address is also specified which is”A1” in this case. In the annotation phase we specify the LSLscripts4 by which the Second Life viewer obtains control of theobjects and performs read/write operation into its communi-cation channel. A sample LSL script for the door exampleis depicted in Figure 5. In order to get refined responseand to provide high flexibility to control the physical objectsour annotation scheme introduces a generic interface. This isespecially important for those devices in which different partsof a 3D virtual device need different annotation rules. Forexample, a table fan have different switch for speed controlwhereas a lamp has different illumination intensity levels or atemperature sensor have different temperature levels. In orderto provide this type of facilities we developed a suitableannotation mechanism that allows annotation into differentparts of the 3D virtual object. In this manner we specify rulesin the XML data that we need to attach with specific regionof the object. This type of multi region annotation scheme isdepicted in Figure 4.

2) Device State Synchronization: In order to render syn-chronized animation in the virtual environment as per the statechanges of the real device, we incorporate state based eventcommunication mechanism. When a user clicks or touchesa 3D object in the virtual environment the system receivesthe event message, which is then saved into an event queue.Further, the system processes the event queue sequentially andtransmits the message respectively to the home automation

4LSL Portal, http://wiki.secondlife.com/wiki/lsl portal

Fig. 5. Code snipped of SL Script to load the XML data to send or receivemessages into the communication channel.

service. At this stage the physical device becomes activatedto handle the input message. In this case if the device failsto initiate its services then an error message is displayed,otherwise it sent ready or Acknowledgment (Ack) message.

In the next step the system sends the actual message, such asopen door, close door, increase light intensity to high, decreasesound system volume to low etc. At this time the smart homeservice receives the message from our developed Second Lifeplug-in. Afterwards, it sends the command message to thespecific physical object X10 or WiFi module at the same timehome service sends to the plug-in to play a synchronizedanimation in Second Life environment that are specified intothe annotated XML file. This type of animation makes thewhole interaction process more realistic for the user. Whenthe physical object tries to close the device or goes to resetit’s state the smart home service receives this updated statesand sends the confirmation to the Second Life plug-in tostop/resume the animation as the physical device changes itssate. This state based interaction scenario are illustrated inFigure 6.

Fig. 6. Proposed system interaction diagram to control a physical device byusing the SL Plug-in.

III. IMPLEMENTATION AND RESULTS

We implemented our proposed system by using MicrosoftVisual Studio 2005 IDE and the primary language used wasVisual C++. In order to develop the Second Life add-on,we locally build the Second Life open source viewer calledSnowglobe5 by using the latest version of CMake6. For thehome automation service we used ActiveHomeScript Libraryand web services dynamic discovery (WS-Discovery) whichwas a multicast discovery protocol defined with the purposeof allowing dynamic discovery and advertisement of targetservices willing to find a specific service queried the networkusing multicast search messages, and services satisfying thequery replied in the process. In our prototype we also usedweb services eventing (WS- Eventing) which was a web serviceprotocol that described how a client (subscriber) registeredwith some events of a web service (event source). Thus,changes in the service were notified to any client withoutrequiring standard polling mechanism.

At the beginning when any device wanted to connect withthe web service, it needed to send multi cast message to searchtarget services by type or within a scope. Then a uni-castresponse was sent to the client as soon as the target servicematched a Probe message. In this way when the connectionwas established successfully then the device needed to sub-scribe to an event in order to send/receive messages to theserver.

We calculated the transmission time from the sender ma-chine to the smart home service by using Equation 1. Whereuser’s average interaction time to interact with the Second Life(SL) viewer was I unit, average data transmission rate via theserver was Π, n was the message size and the time for sendingdata from the receiver machine to the home service was β1unit.

R = I +n

Π+ β1 (1)

After generating an interaction the system approximately re-quired R = (3775 + 270 + 344)ms to complete the transmis-sion. Here, in our experiments the average of the interactiontime was 3775ms, network overhead was 270ms and β1 was344ms.

For the usability test we incorporated the usability evalu-ation guidelines [10] to qualitatively measure our proposedsystem and designed our tests accordingly with the sensoryanalysis of the system involving both the user and the targetedsensory communication. Before performing the usability testwe designed a test plan where we defined our evaluationobjectives, developed questions for the participants, identifiedthe measurement criterion and decided upon the target usersof the system. The test took place at our university laboratorywith ten (10) participants comprising of different age groups.Three (3) of the participants were in age group 13-18, five (5)of them were in age group 18-36 and the rest two (2) were in

5Snowglobe, http://wiki.secondlife.com/wiki/Snowglobe6CMake, http://www.cmake.org/cmake/resources/software.html

age group 36+. In our usability test four (4) of the participantswere female and the rest of were male participants.

TABLE IUSABILITY TEST QUESTIONS TO THE USER.

# QuestionQ1 Perceived system response was acceptableQ2 The device response were realistic and/or acceptableQ3 Consider using the system in 3D Virtual Environment like Second

LifeQ4 Perceived delay between real object response and animation

rendering was tolerableQ5 Easy to get familiar with

The average of the responses of the users were calculatedin percentage form and measured after the usability tests. Theuser responses are shown in Likert Scale [11] in Figure 7. Theratings of the questionnaire were in the range of 1-5 (the higherthe rating, the greater was the satisfaction). The average of theresponses of the users were calculated in percentage form andmeasured after the usability tests. Figure 7 shows the user’sresponses for each given assertions. It is worth mentioningthat more than 80% of the users liked to control their physicalspace using the enhanced system through virtual environment(in this case, the Second Life). 10% of the users liked to usethe system through their mobile devices instead of from thePC. Overall the users were also satisfied with the synchronizedanimation and device responses and 75% of users consentedto that.

Fig. 7. Usability study of the Second Life home automation interactionsystem.

IV. CONCLUSION

In this paper we presented a Second Life prototype systemto monitor and to control a physical home and to bridge the in-teraction gap between the virtual and real world device controlmechanism. The developed system worked as an add-on andloosely coupled to the Second Life viewer. The animation anddevice control data were annotated in the virtual 3D object inSecond Life. The 3D object representing a real device receivedinputs when interacted by a user and automatically respondedwith the state changes in the physical environment. We pre-sented the implementation details of a preliminary prototypeexploring the aforesaid Second Life based interactions in areal-virtual collaborative environment. In future we want toincorporate sound and gesture based device control mechanism

to provide intuitive interaction capabilities to the user in orderto effectively control the Second Life viewer.

REFERENCES

[1] L. Borodulkin, H. Ruser, and H.-R. Trankler, “3d virtual ”smart home”user interface,” in IEEE International Symposium on Virtual and Intel-ligent Measurement Systems VIMS ’02, 2002, pp. 111 – 115.

[2] P. Rashidi and D. Cook, “Keeping the intelligent environment residentin the loop,” in 2008 IET 4th International Conference on IntelligentEnvironments, 2008, pp. 1 –9.

[3] S. Fleck, F. Busch, P. Biber, and W. Straber, “3d surveillance adistributed network of smart cameras for real-time tracking and its visu-alization in 3d,” in Computer Vision and Pattern Recognition Workshop,2006. CVPRW ’06. Conference on, 2006, pp. 118 – 118.

[4] I. O. Sebe, J. Hu, S. You, and U. Neumann, “3d video surveillance withaugmented virtual environments,” in First ACM SIGMM internationalworkshop on Video surveillance, ser. IWVS ’03. New York, NY, USA:ACM, 2003, pp. 107–112.

[5] C. Youngblut, “Educational uses of virtual reality technology. technicalreport no. ida document d-2128,” Institute for Defence Analyses., Tech.Rep., 1998.

[6] B. Zhang, P.-L. P. Rau, and G. Salvendy, “Design and evaluation ofsmart home user interface: effects of age, tasks and intelligence level,”Behaviour and Information Technology, vol. 28, no. 3, pp. 239–249,2009.

[7] C. Thompson and F. Hagstrom, “Modeling healthcare logistics in avirtual world,” Internet Computing, IEEE, vol. 12, no. 5, pp. 100 –104,2008.

[8] S. A. Hossain, A. S. M. M. Rahman, and A. El Saddik, “Interpersonalhaptic communication in second life,” in Haptic Audio-Visual Environ-ments and Games (HAVE), 2010 IEEE International Symposium on,2010, pp. 1 –4.

[9] A. S. M. M. Rahman, S. A. Hossain, and A. El Saddik, “Bridgingthe gap between virtual and real world by bringing an interpersonalhaptic communication system in second life,” Multimedia, InternationalSymposium on, pp. 228–235, 2010.

[10] D. Chisnell, “Usability testing: Taking the experience into account,”Instrumentation Measurement Magazine, IEEE, vol. 13, no. 2, pp. 13–15, 2010.

[11] R. Likert, “A technique for the measurement of attitudes,” Archives ofPsychology, vol. 140, pp. 1–55, 1932.