1

Emotion Sensing Smartphone 12.6.2015

To: Dr. Regina Dugan

By: Michael Rosenfield

Abstract Application developers and content distributors, such as Youtube, rely on historical data and “like buttons” to predict what a user wants to watch next. Unfortunately, because recommendation algorithms rely on this proxy data, results are not satisfactory. The paper proposes the integration of emotion sensing, such as a GSR sensors, into a smartphone and then securely sending the data to the foreground application. The app’s recommendation algorithm will respond with much more relevant recommendations. To limit the scope of the project, the plan is to develop a smartphone case which houses the sensors and communicates with a smartphone through bluetooth. Once prototypes are developed, experiments will be conducted to determine the benefit of the emotional data being used within recommendation algorithms.

Table of Contents Section

1. Introduction 2. Project Description……………………….……….……….……………………... 3. Literature Review………………….………..……….……….……….…………... 4. Plan of Work……………………….…….……….……….……….……………... 5. Work Schedule……………………………….……...……….……….………….. 6. Researcher Qualifications…………………...……………….……….….………... 7. Budget……………………………………………………………….…………. 8. References……….……….……….……….……….……….……….…....……....

Page # 2 2 3 5 6 6 7 8

2

Introduction: There are currently two billion smartphone users worldwide and the average user spends 9 ½ hours on the device every week. A significant portion of smartphone usage is media consumption, including watching videos, browsing the web, and exploring social media. One of the largest challenges for application developers is content recommendations and discovery. The reason this is difficult is because the applications must rely on two main types of data. The first type of data is historical data, such as what videos a specific user has chosen to watch in the past. The second type of data is surveys that require participation from the user, such as clickings thumbs-up or thumbs-down on a video. Consequently, the weak proxy information that is used in current recommendation algorithms, renders the results futile. Because of this, the addition of emotional state data, a better proxy, would improve content recommendations and discovery.

Project Description I propose to research an implementation of emotion sensing technologies, specifically Galvanic Skin Response (GSR), into a smartphone to increase the accuracy of content recommendations. A GSR sensor is capable of reading a person’s skin conductance, which is directly correlative to his or her emotional activity. Through bluetooth, the raw data will be sent to the smartphone, converted into usable emotional data which is then passed to the application on the phone. An application developer can securely use these measurement, in real-time, to recommend content. To limit the scope of the project, I plan to develop a smartphone case which houses the sensors and communicates with a smartphone through bluetooth. (Figure 1)

3

Literature Review

Current Methods of Recommendations: Whether it is for a news app or a video sharing app, most modern recommendation systems are very similar. Due to its complex nature, a prime example of such a system is YouTube. To recommend new content YouTube relies on user activity: the videos the user watched, favorited, and liked. The videos that meet this criterion are inputted into a recommendation algorithm that creates a node for each video and builds branches of other videos that relate to the seed videos. [1] Once the hundreds of recommended videos are created, the system tries to rank the videos using properties that are specific to the user such as how many times the user watched the video and how much of the video they watched. [2] These factors are effective, but it is common for a user to have watched a video completely, yet completely dislike it.

Galvanic Skin Response Sensor: Because view count and time watched are extremely indirect proxies, I propose the addition of a better proxy of user enjoyment: skin conductance. A galvanic skin response sensor is made up of two capacitive “touch points” that are connected by a voltage divider. When a person places two of their fingers on the touch points, their body acts like a large resistor as the electricity is passing from finger to finger. This resistance is a factor of how sweaty the user’s fingers are, as the more sweat on one’s finger, the more conductive the finger is. Using the voltage divider, we can calculate the exact resistance of the body and therefore measure how much sweat the user is producing. [3] Studies have shown that that skin conductance is directly correlated to emotional activity. [4] Due to the small circuitry required to build a GSR sensor, it will not be difficult to fit it in either a smartphone body or a smartphone case. [3]

There have been studies conducted in the past that use GSR in consumer products to measure emotional activity. (Figure 2) For example, researchers at the Blekinge Institute of Technology performed a study where they implemented emotion sensors near the triggers on a video game controller. The study used galvanic skin response sensors as

4

well as electroencephalography, electrocardiography, electromyography and eye tracking equipment to measure how involved a player is when playing in a videogame. The researchers noted that GSR sensors are capable of providing enough information to detect a user’s level of arousal, which can be correlated to a user’s level of immersion. [5]

Using the Data: A key aspect of the project is using the emotional data to benefit both the user and application developer. A study run by researchers at the University of Maryland showed emotional data, collected using GSR, could be used as a means to recommend videos to unique users. The study implemented a system that converted the emotional data into an “emotion status” ranging from disturbed to amused. The study concluded that the optimal system uses a decision tree machine-learning algorithm, which is an algorithm that uses past emotional data to create a mood classification. [6] By implementing this same algorithm, I could convert raw GSR data into simple mood classifications that is sent to the foreground application. This will ameliorate the security connotations that arise with reading emotional data, as the application is limited to reading emotional state. Another study showed that, by correlating emotional state to specific music scores, one could recommend music suitable to a user’s current emotional state. This was done by grouping music chords with identified emotions. [7] The study shows yet another example of improving a user’s experience using emotional data.

Significance of the Project: Though this research involves a single emotion sensor (GSR), research has shown that by combining sensor data from eye trackers, EKG, EEG, accelerometers, and others, one can create a phenomenal emotional model. [8] This emotional model will be tremendously useful if read from modern smartphones. When using a phone, one is often consuming content. The majority of that content is driven by recommendations derived from bad proxy data such as view count, one’s friend’s activity, or how long one spend watching a video. By integrating emotional data into the recommendation algorithm, suggestions will be significantly more tailored to individuals. Users will be presented with content that they want to consume. With emotional data, application developers will also benefit: click through rate will surge and users will spend more time on one’s application. Both of these attributes will grow revenue and increase customer satisfaction. [9]

5

Plan of Work In order to complete my research, I have split the work into three phases. Phase I is building the prototype, Phase II is developing the algorithm, and Phase III is conducting the study.

Phase I: Build The Prototype During Phase I, I will purchase the necessary circuitry to build three galvanic skin sensors that communicate with a smartphone and are all housed within a 3d printed case. This will require basic circuit components, as well as low cost Arduino boards that have integrated bluetooth. Following the circuit design of a GSR sensor, I will build three simple GSR kits that I can place in different enclosures. Next, I will use a 3d printer to design three different smartphone cases with unique conductive sensor placement. I will lastly integrate the GSR kits into the distinct cases.

Phase II: Develop the Algorithm During Phase II, I will build on current recommendation algorithm approaches, but with the addition of emotional data. I will follow the work of YouTube and Amazon, using a deep neural network. In addition, I will develop a channel to allow the smartphone case to communicate with the foreground application on Android. The best way to enumerate the emotional data will be discovered during this research.

Phase III: Conduct Experiment By Phase III, I will have three working prototypes, each with different GSR sensor placements. During Phase III, I will conduct user studies to determine whether access to emotional data increases the quality of recommendations and which sensor placement is most effective. The studies will be conducted by finding 100 participants and letting them use my demo video application. After a some time, I will show recommendation results returned by conventional algorithms and recommendations returned using emotional data. Once I have collected feedback from all participants, I will compile a final report containing the results.

6

Work Schedule Overall, the whole project is estimated to take about ten weeks with the deliverables being three working prototypes, a working emotion integrated recommendation algorithm, and a final report. At the end of each week, I plan to devote a few hours of my time to write status reports that will be send to all mentors of the project, including yourself. This timeline takes into account the roadblocks that are likely going to arise throughout the process.  

Task 1 2 3 4 5 6 7 8 9 10

Source and Purchase Circuitry                    

Build Prototype                    

Develop Recommendation Algorithm                    

Recruit Study Participants                    

Collect, Analyze, and Report Experimental Data

                   

Researcher Qualifications As a third year computer science student at Case Western Reserve University, I have gained the necessary academic knowledge to accomplish this research. Over the past two years, my relevant coursework has included Introduction to Programming in Java, Introduction to Material Science, Introduction to Data Structures, Software Craftsmanship, Computer Networks, Programming Language Concepts, Logic Design and Computer Organization and Introduction to Database Systems. Each of these courses have given me insight and breadth in the world of computer science and engineering. Combining knowledge from each of these classes will allow me to personally create the prototype hardware, as well as develop an exemplar recommendation algorithm.

In addition to the academic background, I have also gained tremendous experience in the technology sphere with my past internships at Google and Motorola. Most relevant to this research, I interned on the Android team at Google where I worked with prototype

7

smartphones. I also internet within the ATAP team at Motorola, where I gained experience developing biometric technologies in the mobile space.

If any problems arise or gaps of knowledge appear, I have the tools and drive to tackle anything necessary.

Budget Due to the simplicity of the electronics and the software focus, the project will cost no more than $5000. The following table describes the few items that will need to be purchased to complete the research.  

Android Smartphone $500

3D Printer $1000 - $3000

Galvanic Skin Response Electronics [x3] $50

Arduino [x3] $100

TOTAL $1650 - $4650

Conclusion Dr. Dugan, well aware of your busy schedule, I will be sure to keep your responsibility on the project to a minimum. As stated above, I will send you weekly status reports and any feedback or ideas will be welcome. Though, if I run into any difficulty, I might need your expertise in product development and research to overcome them.

If all goes well, your involvement will most likely grow as we think past the prototype and towards integrating the research into a consumer product.

8

References [1] S. Baluja et al., "Video suggestion and discovery for youtube," Proceeding of the 17th international conference on World Wide Web - WWW ’08, 2008.

[2] J. Davidson et al., "The YouTube video recommendation system," Proceedings of the fourth ACM conference on Recommender systems - RecSys ’10, 2010.

[3] M. V. Villarejo, B. G. Zapirain, and A. M. Zorrilla, "A stress sensor based on galvanic skin response (GSR) controlled by ZigBee," Sensors, vol. 12, no. 12, pp. 6075–6101, May 2012..

[4] S. Khalfa, P. Isabelle, B. Jean-Pierre, and R. Manon, "Event-related skin conductance responses to musical emotions in humans," Neuroscience Letters, vol. 328, no. 2, pp. 145–149, Aug. 2002.

[5] L. Nacke and C. A. Lindley, "Flow and immersion in first-person shooters," Proceedings of the 2008 Conference on Future Play Research, Play, Share - Future Play ’08, 2008.

[6] X. Y. Chen and Z. Segall, "XV-Pod: An emotion aware, Affective mobile video player," 2009 WRI World Congress on Computer Science and Information Engineering, 2009.

[7] M.-K. Shan, F.-F. Kuo, M.-F. Chiang, and S.-Y. Lee, "Emotion-based music recommendation by affinity discovery from film music," Expert Systems with Applications, vol. 36, no. 4, pp. 7666–7674, May 2009.

[8] J. Gonzalez-Sanchez, M. E. Chavez-Echeagaray, R. Atkinson, and W. Burleson, "ABE: An agent-based software architecture for a Multimodal emotion recognition framework," 2011 Ninth Working IEEE/IFIP Conference on Software Architecture, Jun. 2011.

[9] G. Linden, B. Smith, and J. York, "Amazon.com recommendations: Item-to-item collaborative filtering," IEEE Internet Computing, vol. 7, no. 1, pp. 76–80, Jan. 2003.

Figure 2: http://www.element14.com/community/servlet/JiveServlet/showImage/38-4762-46098/MakeArticleFig01_02.png