Using Metrics from Mobile Technology to Improve Self- Awareness …media.virbcdn.com/files/77/9e0510b2f513fe72-FinalReport... · 2016-05-07 · HCI 594 Final Project Report Team:

HCI 594 Final Project Report Team: 2

Winter 2016 Date: March 13, 2016

Winter 2016 - Page 1

Using Metrics from Mobile Technology to Improve Self-

Awareness of Goals Actualization

FINAL PROJECT REPORT

Jason Boggs

Leilani Johnson

Lindsay Jones

Cameron Miller




Table of Contents

Abstract ................................................................................................................................................. 3

Introduction ......................................................................................................................................... 3

Project Goals ......................................................................................................................................... 4

Methods ................................................................................................................................................ 5

Competitive Evaluation ............................................................................................................................ 5

Diary Study and Interviews ...................................................................................................................... 6

Cognitive Walkthrough ............................................................................................................................. 6

Assessment Test ....................................................................................................................................... 7

Results and Findings ............................................................................................................................ 7

Competitive Evaluation ............................................................................................................................ 7

Diary Study and Interviews ...................................................................................................................... 9

Information Architecture ....................................................................................................................... 11

“ScreenTime” Prototype ......................................................................................................................... 12

Cognitive Walkthrough ........................................................................................................................... 13

Assessment Test ..................................................................................................................................... 17

“ScreenTime” Prototype: Revised .......................................................................................................... 18

Conclusion .......................................................................................................................................... 19

References .......................................................................................................................................... 20

Appendix A: Competitive Evaluation ............................................................................................... 21

Appendix B: Diary Study & Interviews ............................................................................................. 23

Appendix C: Journey Map .................................................................................................................. 30

Appendix D: Personas ........................................................................................................................ 31

Appendix E: Diary Study Spectrums ................................................................................................ 32

Appendix F: Prototype ....................................................................................................................... 33

Appendix G: Cognitive Walkthrough ............................................................................................... 37

Appendix H: Assessment Test .......................................................................................................... 43




Abstract Today, everyone and everything is connected. Many of us feel compelled to be in constant contact with our mobile phone and other devices. This attachment to technology may lead to individuals devaluing their personal relationships and reducing their ability to achieve their goals (Rainie & Wellman, 2012). Our project focused on providing people with an opportunity to understand the relationship between phone usage and productivity in order to ultimately improve their lifestyles. While there are many existing apps in the market space that provide device usage data, our app goes one step further by adding relevancy; it demonstrates that the same idle time spent on our devices could be used toward completing short-term goals. It also provides comparison data so that users can review their usage against their friends and those with similar characteristics as them. To better understand our users’ needs, we conducted several evaluations including a competitive analysis, diary study, interviews, and two usability tests. In this report, we provide a review of our methods and findings, present our app ScreenTime, and offer suggestions for future research.

Introduction

At the turn of the 21st century, technology advancement and the availability of information became the focus of many inventors and businesses, morphing our lives into what we know today. In 2016, everyone and everything is connected. So much so, that not only is it an expectation, but it is also a need for many to constantly stay in tune with many different social groups. In their book Networked: The New Social Operating System, researchers Lee Rainie and Barry Wellman (2012) call this phenomenon “networked individualism,” where humans are no longer restricted to socializing within tightly knit groups, yet have to maintain how these groups overlap and maintain connections. Consequently, being this attached to technology may pose several challenges. That is, people are likely missing out on present experiences and actually devaluing their relationships by always needing to tap into other social spaces on their devices (Lee & Park, 2011). In addition, the amount of time that many people spend using technology is likely inhibiting them from achieving their goals or completing important tasks (Newport, 2015). Additional review of the literature revealed that people significantly respond to text-based motivational cues regarding their activity (Tabak, Akker, & Hermens, 2014). Furthermore, by engaging users with notifications based on how their phone usage may be affecting the actualization of their goals, we believe that we will be supporting long-term behavioral changes, as discussed by Azar et al. (2013).

This project focuses on providing people with an opportunity to understand the relationship between phone usage and productivity in order to improve their lifestyles. We want to design a mobile app that provides details on device usage and how that same amount of time could be used to accomplish a user’s daily tasks. While other apps provide device usage data, our app intends to go one step further by adding relevancy--by helping people better understand their device usage habits and how to alter their behavior to focus more on achieving their goals towards a more balanced lifestyle.




The target customers for this experience are avid mobile technology users who are interested in improving their time management skills and managing a balanced lifestyle. Ages can range anywhere from 16-65 years old. Users would install the app on their tablet or smartphone to capture device usage details such as browsing time on phone, length of app usage, home button taps, average battery life, text and call activity, and more. A user could then use this information to better assess how they actually want to be connected to their devices moving forward via predefined goals.

For example, Sarah has one week to read three chapters of her biology book and study for her first midterm. Sarah already knows that she tends to get distracted by her social needs and finds that her devices and ability to connect can make it even harder to focus. Her primary goal this week is to study for her biology exam, whether that is using offline or online materials. By the end of the week, Sarah spent 9% of her awake time using the Facebook app, 6% using Snapchat and 4% scrolling through and posting pictures on Instagram (assuming she logged 8 hours of sleep a night). That adds up to 19% (or about 21 hours) of her time focused on social activities rather than aligning to her original goal of studying that week. She also hit the home button on her iPhone to “check it” for incoming calls, time and texts on average 16 times an hour. Sarah knew she had spent time socializing on her devices, but she didn’t realize how much it had taken from her study time. With this knowledge, the app provides a few recommendations on how Sarah can shift her behavior to get in the study time she needs. Sarah reassesses how she wants to spend the following week on her devices to improve her grade for her next exam.

Project Goals Table 1 compares the goals that we originally envisioned in our Project Proposal to our actual goals. We adjusted all of our goals as the project evolved.

For our first goal, identifying the types of phone usage data people would find valuable to track, we removed the survey responses as a method because we realized that the survey wouldn’t be the right activity to obtain this type of information. We decided that reviewing the diary study responses would suffice.

We liked our second goal, identifying the types of usage required to affect user behavior, but decided to postpone it because we wouldn’t be able to obtain this information from our existing methods, and we did not have enough time to introduce another method.

Similarly, we realized after completing the diary study that we needed to refine the third goal, identify the magnitude by which phone usage metrics impact the completion of goals, to say that we would like to identify the perceptions people have on their phone and the implied subsequent behavior. We did not have enough time to study impact, but we did have enough time to assess implications. We added the qualitative data from the interviews as a measure.

We changed the fourth goal from “devise a blueprint” to “prepare a set of feature and data metric recommendations” for clarity. We also added the assessment test as one of the two methods to measure this goal.




Table 1: Proposed vs. Actual Goals

Goal # Proposed Goal Actual Goal

2.1 Goal: Identify the types of phone usage data that people would find valuable to track.

Measure: Survey and diary responses related to value of phone usage.

Goal: Identify the types of phone usage data that people would find valuable to track.

Measure: Diary study responses related to value of phone usage data.

2.2 Goal: Identify the types of phone usage data required to affect users behavior.

Measure: Qualitative data from surveys, interviews and diary study.

postponed to future research

2.3 Goal: Identify the magnitude by which phone usage metrics impact the completion of goals.

Measure: Qualitative data acquired from diary study.

Goal: Identify the perceptions people have on their actual phone usage and the implied subsequent behavior.

Measure: Qualitative data acquired from diary study and interviews.

2.4 Goal: Devise a blueprint for implementing this data into an app.

Measure: Expert evaluation of low-fi prototype.

Goal: Prepare a set of feature and data metric recommendations for a goal actualization mobile app.

Measure: Assessment and expert evaluation of low-fi prototype.

Methods

Competitive Evaluation Each group member reviewed a mobile app that provides tracking of mobile metrics. Apps reviewed included Moment, Break-Free, QualityTime, and Forest. The group members collected the following data types into a table: name of app, available operating systems, app store rating, user comments, collected metrics, and how the app has made the data meaningful. “Meaningful” was determined by how the data was displayed in graphs, grouped into concepts people are familiar with, and if the metrics were used with actionable functionality such as notifications and disabling apps. The group met to analyze the data and compare the information to determine which app to use for the diary study and interview. This method did not undergo any major revisions based on the early feedback provided by class peers and our advisor.




Diary Study and Interviews In order to understand people’s impressions of data metrics apps, our group implemented a combined diary study and interview. We chose the Moment app as our stimulus based on the competitive evaluation data. We had nine participants (six females, three males) from 25 to 60 years old.

For the diary study, moderators held an initial meeting with participants either online or in person to sign a consent form and set up Moment. Participants were asked to submit a diary entry once per day over the course of seven days via Google Form. The form was used to collect total screen time, number of pickups, longest time spent on their phone in one sitting, locations where the phone was used, and thoughts of the participants. See Appendix B: Diary Study & Interviews Test Plan.

After the diary study period ended, moderators met with each participant either online or in person and completed a closing semi-structured interview which lasted approximately 30 minutes. The interviews were audio recorded to assist with capturing the responses of participants.

The quantitative data was analyzed using pivot tables to compute the averages of the pickups, hours, and longest session time over the seven days of the diary study. The averages were used to create spectrums for the three categories. In addition, the group completed a closed card sort analysis of the qualitative data using Trello. Quotes were organized into several categories:

● Moment Pain Points ● Desired Features ● Things I like about Moment ● Why would I keep using Moment ● Why wouldn’t I keep Moment ● Initial perceptions of my phone usage ● Importance of a “Charged” Phone ● Changes that the participant would make

To support Goal 2.2 - identify the types of phone usage data to affect behavior - this portion of the study originally included an interview survey to be distributed to participants one week after their interview. We removed this method since Goal 2.2 was postponed to further research.

Cognitive Walkthrough Our group conducted a cognitive walkthrough to evaluate our first iteration of the prototype with the assistance of our class peers, who we identify as usability experts. Since the prototype used in this evaluation was designed specifically for the asynchronous nature of the assessment, the implementation included self-guided components such as an index page with a list of links to begin each task and a confirmation message at the end of each task. The experts provided their input into a Google Form, which was divided by the five tasks and employed the same set of questions in a “Yes-No, Explain” format, with the additional capability to upload screenshots to provide additional context. The exact steps to complete each task were provided to ensure experts understood the intended process. After completing the walkthrough, six experts provided




feedback on the process and the results in D2L. The data from both the Google Forms and D2L was placed into a Google Doc for further analysis with the assessment test. See Appendix G: Cognitive Walkthrough Test Plan.

Assessment Test The assessment test ran in parallel with the cognitive walkthrough and focused on collecting feedback from non-experts in the field of user experience. Four participants (two male, two female) between the ages of 18-40 were recruited to evaluate the first prototype using the mobile device of the moderator. Since this was an in-person test, we did not include the self-guided components used for the cognitive walkthrough prototype; the prototype was modified to allow participants to explore the app. The participants were asked to complete five tasks while thinking aloud during the process. They were then asked five follow-up questions about their experience. The testing was audio recorded to assist with capturing the responses of participants. See Appendix H: Assessment Test Plan.

The quantitative data was placed into a Google Spreadsheet, and the qualitative data was placed into the same Google Doc as the cognitive walkthrough data. Our group met online to review the collected data from both tests to identify and prioritize the needed changes to the app.

Results and Findings

Competitive Evaluation For the competitive evaluation, we discovered several differences and similarities between each app. Table 2 shows a breakdown of operating support by app. Forest was the only app that was available on Windows and Desktop. Each app was available on Android except Moment, and each app was available on iOS except QualityTime. We realized through our evaluation that only a few apps provided a breakdown of time spent per app. For example, a user could see that out of the one hour that they spent on their phone, thirty minutes was spent using the Facebook app, and thirty minutes was spent using the CNN app. We learned from QualityTime’s FAQ page that Apple (iOS) did not support one app having the ability to record the activity of another app. As a result, apps such as Break-Free provided versions of the app with and without this feature for Android and iOS respectively, whereas other apps like QualityTime only offered an Android version with this feature. This feature became a sticking point for us when determining which app to chose for our study. As we learned from our diary study participants who are iOS users, many users desired to see this breakdown.

Table 2: Operating System Support by App

App iOS (Apple) Android OS Windows Desktop

Moment ✓




Break-free ✓ ✓

QualityTime ✓

Forest ✓ ✓ ✓ ✓

Table 3 is a summary of the available features, including time management tools and incentives to encourage behavioral change. Moment, Break-Free, and QualityTime provided the metrics in hourly (timeline), daily, and weekly views, and they allowed the user to restrict their daily use with reminders as they’re approaching their limit. Break-Free and QualityTime allowed users to set up a “do not disturb” period, which included directing calls to voicemail, auto-responding to texts, and disabling other apps. Moment and Break-Free gave the user a daily score and a color rating (red, yellow, green) based on use. Break-Free also incentivized decreased phone use by offering rewards and badges.

While Forest did not provide any of the features stated above, it was the only app that included a gamified experience to encourage behavioral change. In it, users would plant a tree when they needed to focus on a task that didn’t involve using the phone, such as studying. Should the user exit out of the app in order to use another app, the tree would die.

Table 3: Operating System Support by App

App Provide multiple views of data

Restrict daily use

Provide daily score

Offer reward Institute “do not disturb” period

Provide gamification

Moment ✓ ✓ ✓

Break-free ✓ ✓ ✓ ✓ ✓

QualityTime ✓ ✓ ✓

Forest ✓

Each app had a score in the Apple and/or Android app store. Break-Free (Android version), QualityTime, and Forest received at least four out of five stars. Comments were mostly positive, though there were some concerns around the app’s access to private data and the excessive drain on the battery. The iOS version of Break-free received the lowest score with a 2.2 due mostly to complaints around glitchiness and battery drain.

In summary, we found that Moment, Break-Free, and QualityTime offered very similar features, including usage metrics and available time management tools. While Forest did not provide any of these features, the gamification component offered another approach towards changing user behavior. See Appendix A: Competitive Evaluation.




Diary Study and Interviews During the diary study, we measured the participant’s total daily time spent, total daily pick-ups, and longest time spent on their phone in one sitting per day and created spectrums from this data (See Figure 1). We shared the individual results with our participants during the interview to gauge their reaction. See Appendix B: Diary Study & Interviews.

Five of the nine participants spent an average of two to three hours on their phone per day. Participant 6 (P6) spent the most time on their phone, averaging four to five hours per day and more than six hours on two days during the seven-day period. This participant shared during the interview that while they would have expected to average only three hours a day, they understood why the numbers were so high since they listened to music during their commute and at work.

In fact, when asked about their perceptions of their phone use, many participants admitted that they were initially shocked about their numbers but then rationalized it as they considered the many uses of their phone. Some participants even began to question why there is a negative association with increased phone use. As P3 stated, “It’s not just texting and talking, that’s where I get my everyday news. So I don’t think I should feel that bad about being on it so much.”

P6 also had the highest number of pickups along with P4, averaging 61 to 80 times per day. P4 was surprised to learn that they picked up their phone 96 times in one day. Four participants picked up their phones 41 to 60 times a day. Many of them expressed that picking up their phone was an automatic habit, one they did not realize they were doing so much.

P5 spent the longest time on their phone in one period, averaging 41 to 50 minutes per day and spending more than two hours on their phone in one sitting on three of the seven days. They shared during the interview that most of this time was using the GPS on their phone as they were traveling.

When asked about most liked features, participants commented on the simple design, the easy-to-read data, as well as seeing the phone use breakdown by location. Regarding least liked features, two participants mentioned that the app drained their battery. Six participants suggested that the app should provide a breakdown of time spent per app. Three participants challenged the accuracy of the usage tracking or admitted that they did not understand what the metrics meant. For example, P6 did not know whether a “pickup” meant that they had pressed the phone’s home screen, or that they had unlocked their phone. Either way, they felt their pickup numbers were inaccurately high.

Some participants suggested that having comparative data would help them interpret their results. P1 commented, “I think it would be interesting if Moment could report national averages on what it's collecting so you could compare.” P7 asked whether the moderator had a benchmark for what’s normal, stating that “I would hope that [my number] is lower than people my age.”

Participants were unsure if they would keep Moment after the study. Half of the participants said that being aware of time spent is interesting, but not enough to keep the app. P7 noted, “I don’t know [if I would keep Moment] because I don’t think I’d check it that often. [...] I wouldn’t monitor it the way that I monitor my steps.” P3 agreed, stating that, “It was interesting for the week that I watched it, but I don’t feel the need to really use it.”




Figure 1: Spectrums

In review, participants liked the Moment app design and the general concept of a usage tracking tool. However, they wanted a better understanding of what and how information was being tracked. Likewise, they desired the ability to exclude certain apps from being tracked, such as their GPS and music. Since the app was running in the background constantly, it had the potential to drain the battery more so than other apps. There was also a great desire to see phone use broken down by app. While we had learned through our competitive evaluation that app tracking is not available on iOS, we took note of the importance of this feature for users.

The features stated above are straightforward considerations, but this activity exposed two fundamental challenges with these types of apps. First, participants need context in order to make sense of their data. Knowing that they picked up their phone 75 times was meaningless unless they knew that another individual picked up their phone 20 times. Second, the Moment app lacks longevity on a user's mobile device because participants did not know what to do with the data. The participants saw that they picked up their phone 75 times, but they did not know how it was meaningful. As P6 stated, “the app doesn’t make the data functional, [it’s] just a tidbit or stat.”




Information Architecture After the competitive evaluation, diary study, and interviews, our team completed a series of activities to inform the design of our interactive prototype app.

PERSONAS

We started with two personas. Our first persona, Marcus, is a 40-something contractor who uses his phone for long periods of time to get things done. Being a heavy phone user, he wants to see his daily and weekly phone use but with the option to exclude apps that shouldn’t count towards his total usage. He relies on his phone all day, so having a strong battery is critical. Our second persona, Unnati, is a small business owner in her 30’s. Her phone use can be described as sporadic; she checks her phone when she needs a short mental break or a distraction. She is interested in getting as much detail on her phone use as possible: weekly and daily views, by location, and by app. See Appendix D: Personas.

FEATURES MATRIX

From there, we created a features matrix as shown in Figure 2. This matrix is a list of possible features for our app based on interview and diary study input, including tracking specific app activity, number of pickups, and a social feature to compare data. We then reviewed each feature and assigned a weight based on our personas’ needs. For example, while setting daily limits would be important to Marcus but not important to Unnati, both personas would need to see their activity by app. We came up with a total of seventeen features, nine of which would be essential to our app, five that would be nice to include, and three that would not be important to implement.

Figure 2: Features Matrix




JOURNEY MAP

Next, we created a journey map to draw out the user flow based on these features. And as the final step, we created a quick sketch of the homepage/dashboard. See Appendix C: Journey Map.

“ScreenTime” Prototype We built a medium fidelity interactive prototype using Axure based on the Information Architecture activities stated above that would be used for the cognitive walkthrough and assessment test. The prototype can be found via AxShare here: http://6ft4yt.axshare.com

The first iteration of the prototype employed a variety of common design patterns. A bottom menu bar was utilized for primary navigation and included Other People, Goals, Home, Limits, and ScreenTime. “Home” corresponded with a dashboard where the user could view daily stats regarding phone usage. “Other People” corresponded with a page where a user could view aggregated statistics of others’ phone usage, including their friends. This was designed to help the user gain perspective on what may be considered “normal” usage. “Goals” corresponded with a page where the user could “teach” the app about what he or she desires to accomplish. “Limits” corresponded to a page where the user could set a hard usage limit on a specific app. “ScreenTime” corresponded with a list of previous days and statistics. The user could drill down to a specific day to see more details regarding their usage.

Figure 3: Prototype Screenshots (the Home screen and a specific day’s timeline)




Cognitive Walkthrough Usability experts were provided a task-based version of the initial prototype for their evaluation which included a list of the steps to complete each task. For each task assessed by experts, we collected data related to the difficulty of each task on a Likert scale of 1 to 5. We also asked experts to provide “Yes/No” responses to five questions to evaluate the effectiveness of each task. See Appendix G: Cognitive Walkthrough for additional tables.

Table 3: Feedback from Experts for Task 1 - “Sign-up for the ScreenTime app”

E1 E2 E3 E4 E5 E6

Difficulty Rating (1 easy - 5 hard) 1 2 1 1 1 1

Will user notice the correct action is available? Y N Y Y Y Y

Will the user associate the correct action with the effect to be achieved?

Y Y Y Y Y Y

Will the user try to achieve the right effect? Y Y Y Y Y Y

If the correct action is performed, will the user see that progress is being made toward solution of the

task?

Y Y Y N N Y

TASK 1

Experts considered the sign-up task (Task 1) easy with five experts responding with a 1 for difficulty rating. The sixth expert, Expert 2 (E2), rated the difficulty as 2 (somewhat easy) but explained a user would not notice the correct action, “When I went to the Auto Track feature I tried to slide the ‘On’ slider to the right to turn Auto Track ‘on’, but couldn’t. The visual feature looks like a slider but it’s really clickable.”

All the experts agreed a user of the app would be able to associate the correct action as well as achieve the right effect of making it through the signup process. E3 mentioned “[The app] uses a very familiar process that most app users will recognize. Down to the login with Facebook or Google options.”

Four of the six experts noted the visible progress through the the sign-up task was clear and provided good context. The two remaining experts stated that users would not see progress, pointing to the fact that the username did not show up after registration. E5 recommended that a confirmation pop up should be included as part of the sign-up process, though it was not clear what the confirmation message should consist of.

Overall, the experts responded positively to the first task with only a few suggestions for improvement. A couple of the experts did run into issues using the prototype such as the slider control for turning on auto tracking. Since the prototype was prepared in Axure, the slider control worked as a clickable element and not




as a touch interaction like the slider affords. The experts also recommended strengthening the prototype with respect to validation.

TASK 2

Experts noted more difficulty with Task 2, checking usage for the previous day, with three of the six experts reporting a difficulty rating of 3. A majority of the experts stated the user would not notice the correct action to take. They attributed this to a confusion with the instructions provided to complete the task which asked them to click “ScreenTime”; the wording was displayed as part of the app name at the top of the screen as well as an option in the bottom left of the screen (see Figure 4). E2 explained, “...your logo is called ‘ScreenTime’ and the actual task activity is also ‘ScreenTime’. So when you ask me to click on ‘ScreenTime’, I clicked on the logo because it was the first thing I could see.” The experts were split in half regarding whether users would associate the correct action with what was supposed to be achieved.

Figure 4: “ScreenTime” is used in logo and main menu

Even though the experts stated users would not notice the correct action, they all agreed users would try to achieve the right effect. E5 stated that “the correct action looks like a button to be pushed,” while E2 noted, “Well not on the first try anyway...the logo takes you back to the landing page where you can just select the task again.” In addition, five out of the six experts indicated users would see progress as they completed the task.

For this task, the experts identified both an issue with crossed labeling of the app name and menu options. They also mentioned confusion about the last screen of the task, the timeline. E4 stated, “I don’t understand the last page in the task. I don’t think it is clear how this graph represents the usage statistics.”




TASK 3

Experts rated Task 3, view today’s specific app usage and exclude unwanted apps, as somewhat easy on average, with four of the six experts recording a difficulty rating of 2. In spite of this, one-third of the experts did not believe the user would notice the correct action to take. E4 explained “The icon and the name of ‘ScreenTime’ are not clear. It doesn’t explain the purpose of the option.”

One-third of experts also did not believe the user would associate the correct action with the desired effect. E4 referred back to the issue of having the “ScreenTime” as a menu option which could prevent the user from starting the task. E6 wondered, “...is there another way to filter the apps without having to go through that many steps.” Although the experts mentioned these concerns, five out of the six experts responded that the user would achieve the right effect for the task and all six responded the user would see progress towards the solution of the task.

The experts found interacting with the timeline feature the biggest challenge for this task. For the timeline (see Figure 5), the bubbles did not appear to provide the appropriate affordance to lead the user to press them in order to see the app time breakdown. E4 explained, “Also, the circles in the statistic data are not clearly clickable. I didn’t know that they are clickable before I read it in the steps.” E5 also noted, “I did not automatically click the bubbles assuming they would give additional information.”

Figure 5: Timeline view with clickable bubbles




Task 4

Experts indicated that Task 4, setting a goal, was relatively easy to complete, with two experts responding with a score of 1, two responding with a score of 2, and two responding with a score of 3. In addition, five of the six experts indicated users would notice the correct action is available and would associate it with the effect to be achieved. E1 was the only expert to disagree, stating, “The plus button to enter the goal is not intuitive.”

Five of the six experts indicated the user would try to achieve the right effect of adding a goal. This aligns with some of the expert’s feedback statements; E3 stated the task was “simple to understand” while E4 noted, “This task was very easy to accomplish because the steps and the icons make sense to me.” As for the user seeing progress towards the solution, four out of six experts responded that users would see progress towards the solution. E5 was one of the two who indicated ‘No’ and explained that the confirmation text was “Still too quick and subtle.”

The interesting feedback from experts completing this task was the diametrically opposed viewpoints of the experts for different aspects of the task. Experts either stated they liked the goal action icons like the plus icon and indicated they were easy to understand while others stated the icons were not intuitive. The negative feedback caught our team by surprise, as we had modeled the goal interface off of the Reminders app on the iPhone. Experts were also opposed on the overall effectiveness of task; E2 offered the perspective that “[The user] might say this takes too much time. And move on…” and posed the question that the user “might think it’s too much work.” These differences of opinion and the seeming contradictions of finding the task easy but stating the task could use various improvements support the need to review this task further.

TASK 5

Task 5 was marked as one of the easier tasks to complete, with five of the six experts giving the task a difficulty score of 1. E2 was the only expert who did not consider the task easy and gave a difficulty rating of 4. The expert identified their confusion over the labeling of sections like “General”, “Other People”, and “Your Friends”. E2 noted that the users might be challenged with “Understanding the content and what usage data belongs to whom.” In spite of this feedback, E2 and the other experts all responded with ‘Yes’ to all four major questions regarding the user's ability to notice the correct action, associate it with the effect to be achieved, achieve the right effect, and see progress towards the solution.

The feedback of our experts helped us identify the major concern for this first iteration of the ScreenTime app: Communication. Language was a common concern of the experts with terms like “Pickups” and “Other Friends” not conveying our intended meanings of how many times the phone was accessed and all people who use the app including friends, respectively. Missing language was also a concern, with our privacy message for auto-tracking not providing enough information for users to make an informed decision or if they could opt-out and still use the app. Experts noted gaps in the app where confirmations and notifications were missing and could be helpful towards keeping the user informed through the various points in the app.

Communications through affordance were also pain points for the first prototype. The bubbles or circles model used for the app usage timeline were intended to invite users to tap on them, but a set of experts




mentioned that they did not understand at first the intention was for them to tap or click the circles to see more information. The goals section of the app also had an affordance problem; though icons like the plus “+” sign were easy to understand the intended action of adding a new goal, the goal data entry fields and other elements did not convey the intent for users to define their own goals.

After completing the analysis of the cognitive walkthrough, we immediately reviewed the assessment test results. The feedback from our experts and non-experts combined helped inform the next iteration of ScreenTime.

Assessment Test For the assessment test, we asked our non-experts to go through the same five tasks as the cognitive walkthrough but without the guided steps. We measured the participant success rate (Graph A), difficult ratings per task (Graph B), and general feelings about the app.

A task was considered to be successful if the participant was able to complete it with minimal to no assistance from the moderator. A task was considered partially successful if the participant asked one or two clarifying questions or only needed assistance less than once to complete the task. A task was assigned a failure rating if the participant could not complete the defined task on their own or had rely heavily on the moderator for assistance. Based on this criteria, all four participants completed Task 1 (signing up), Task 3B (exclude apps), and Task 5 (compare stats) successfully with minimal-perceived difficulty, an average of 1.4 on a 5-point likert scale. On average, participants were 75% successful in completing all tasks. See Appendix H: Assessment Test.

Moreover, Task 3A (view app usage) had the lowest success rate at 38% and the highest perceived difficulty at 3.67. Assessment Participant 03 (AP03) was the only participant to successfully complete the task and consider it to be very easy. AP01 was partially successful, but considered the task to be very difficult. Both AP02 and AP04 failed the task, but we did not ask AP04 to rate the task since they gave up. Participants had a difficult time finding detailed app usage data and had to dive deep into the app to do so. For example, AP01 mentioned, "It wasn't obvious that you were supposed to hit the dots once you were in ScreenTime, there's not a link or anything".

Task 4 (set a goal) was evenly split. AP01 and AP03 successfully completed the task and AP02 and AP04 failed the task. Despite this, each participant considered the task either very easy or easy, averaging a 1.25 rating. Participants that failed the task confused goal setting with setting limits. AP01 thought they had succeeded, when in fact they never made it to the goals screen. The participant noted, “I would think that any of these apps [in limits section] you would want to limit your usage with any of these apps, you could set a limit for how much you are allowing yourself to be on it."

Lastly, Task 2 (check previous metrics) had a 63% success rate, with two successful completes, one partial success, and one failure. This task was the second most difficult task, averaging a 3.5 difficulty rating. For this task, participants attempted to engage with the core content area of the screen first rather than the horizontal menu bar. There are timeline options at the top of the dashboard screen in addition to the “ScreenTime” option below that participants wanted to engage with first, but could not. For example, AP03




stated, “I would normally select the buttons at the top of the page". In addition, participants stayed away from the “ScreenTime” menu item since it was branded the same as the name of the app and was not simply titled “Screen time”.

The assessment test provided the team with more insight into what improvements could be made, which were consistent with the major themes uncovered through the cognitive walkthrough. We need to ensure that we are using consistent terminology throughout the app and make the usage metrics by app easier to find. Perhaps our biggest takeaway is that while providing the goal-setting features is the core differentiating feature for ScreenTime, the testing showed that our prototype did not help participants make the connection between setting goals in order to spend less time with technology.

Graph A: Task Success Rates Graph Graph B: Participant Reported Difficulty Rating

“ScreenTime” Prototype: Revised As the final step of our study, we revised the ScreenTime app based on the collected feedback from the usability tests. To begin, our group placed the suggested changes in a Google Doc; 12 items in total. We reviewed the list and removed any changes that were not within the scope of a medium fidelity app, such as validating text fields or implementing error messaging. We also removed changes that we could not address within our timeframe. See Appendix F: Prototype Recommended Changes to App.

After voting, we made the following revisions:

1. Change “ScreenTime” menu item to “History” so that it is not confused with the logo. 2. Make “This Week” and “All Week” tabs on the homepage clickable. 3. Create a tutorial upon sign up to explain ScreenTime’s purpose and to demonstrate creating a goal so

that users have a better understanding of the connection between metrics and goal setting. 4. Add the usage by app data to the timeline page so that it is quicker for the user to access. 5. Make the labeling of the “exclude apps” feature consistent (previously interchanged between

“exclude” and “filter”).




We also made minor design improvements throughout. The revised prototype can be found via AxShare here: http://shlaoj.axshare.com/.

Figure 6: Revised Prototype Screenshots

Conclusion This project focused on providing people with an opportunity to understand the relationship between phone usage and productivity in order to improve their lifestyles. We conducted a series of activities to explore the strengths and opportunities of existing metrics tracking apps and learned that people are interested in seeing their phone use data. But without a benchmark of what “normal” looks like or knowing what to do with this information, many questioned the usefulness of these apps.

In response, we developed the app prototype, ScreenTime, which introduced two new features: a goal setting feature that encourages users to improve their productivity by reminding them about short-term tasks they wish to accomplish during their free time, and a benchmark feature that allows users to compare their data with their friends and other users who share similar characteristics. Based on expert and user feedback of our prototype, we improved the usability, most notably adding a tutorial section for goal setting so that users better understand the relationship between spending less idle time on their phones and accomplishing more of their tasks.

If we continued this study, we would like to improve the fidelity of our prototype and then conduct another, longer study to understand two things: (1) the types of data that would most motivate users to reduce time spent on their phone and (2) whether users now understand the value of the goals feature and its relationship with time spent, now that we have revised the app.




References Azar, K. M. J., Lesser, L. I., Laing, B. Y., Stephens, J., Aurora, M. S., Burke, L. E., & Palaniappan, L. P.

(November 01, 2013). Mobile Applications for Weight Management: Theory-Based Content Analysis. American Journal of Preventive Medicine, 45, 5, 583-589.

Chan, M. (2014). Multimodal Connectedness and Quality of Life: Examining the Influences of Technology Adoption and Interpersonal Communication on Well-Being Across the Life Span. Blackwell Publishing Ltd.

Lee, K.C., & Park, B.W. (2011). Ubiquitous Computing and Multimedia Applications. Second International Conference, UCMA 2011, Daejeon, Korea, April 13-15, 2011. Proceedings, Part II.

Newport, F. (2015). Most U.S. Smartphone Owners Check Phone at Least Hourly. Gallup.

QualityTime - FAQ. (n.d.). Retrieved March 13, 2016, from http://www.qualitytimeapp.com/faq/

Rainie, L., & Wellman, B. (2012). Networked: The new social operating system. Cambridge, Mass: MIT Press.

Tabak, M., Akker, H., & Hermens, H. (March 01, 2014). Motivational cues as real-time feedback for changing daily activity behavior of patients with COPD. Patient Education and Counseling, 94, 3, 372-378.




Appendix A: Competitive Evaluation See here: https://docs.google.com/document/d/1iMkVpueolW09kvgETm5Jky-bRdaTUBTlFoveG1ddydk/edit?usp=sharing




Access to Competitors

● Moment: https://inthemoment.io/

● Break-free: http://www.breakfree-app.com/

● QualityTime: http://www.qualitytimeapp.com/

● Forest: http://www.forestapp.cc/




Appendix B: Diary Study & Interviews

Test Plan

Objectives The purpose of this study is to gather users’ impressions regarding mobile applications which collect and display phone usage data. We would like to identify the type of data users would or would not want to be tracked or would want to be informed about. We’d also like to understand the types of goals users would like to achieve in place of the time they spend using their phone. The results of this study will be used to inform the design of our future prototype as well as data and interactions our group should focus on while conducting the diary and interview study.

Research Questions There are several research questions we hope to answer during this study, including:

● Do users set goals/tasks? Are they achieving them? Is time a factor in achieving them? ● How much time do users spend on their phone? ● How do users feel about the time spent on their phone? ● Is there a difference between how much time users think they spend on their phone and the actual

time spent? ● Do users see a relationship between time spent on their phone and productivity? ● What information are users interested in tracking on their devices? ● What is most important to users when connecting with others?

Methodology To answer the research questions above, we will conduct a qualitative and quantitative study with two components: Interview and Diary study.

1. Diary study: We will ask four to eight participants to install a competitor application (Moment) on their mobile devices and record their usage and thoughts over a one week period.

2. Interview: We will interview these participants after the diary study to understand perceived versus actual phone usage, impressions of the types of metrics captured, personal goals, and productivity.

Measures The following chart identifies the measures collected per component.

Diary Study Measures Interview Measures




Daily total screen time Perceived vs actual phone use

Daily total pickups Preference of types of metrics captured

Most time spent in one place Feelings around personal connectedness

Places traveled throughout day Feelings around others’ connectedness

Whether they looked at Moment throughout day

General comments

Procedure 1. Diary study procedure (days 1-7 of study):

a. Introduction

b. Project screening and details

c. Sign consent form

d. Moment installation instructions

e. Daily diary log instructions

f. Wrap-up and interview scheduling

2. Interview procedure (day 8 of study): a. Conduct interview via Moderator’s Guide

b. Provide the ability to uninstall Moment

c. Wrap-up and Thank You

Test logistics

General

● Each team member will contribute as a moderator and will recruit 1-2 participants

● We will test an existing competitor app called Moment within iOS, version 2.4.1a.

● Participants must have an iPhone using iOS 9 or later and be willing to install Moment on their phone.

● Participants must allow the app to auto-track, use their location data and run the app in the background through the duration of the study.

● Participants will not be compensated for their time.

● Data collected will remain confidential.

● Participants are expected to complete a daily questionnaire to log their Moment data.




● Researchers will send daily emails with the questionnaire link for participants to log data from the previous day.

Diary study

● Participants will receive verbal or electronic instructions on how to begin and complete the study.

● Participants will be required to complete a 5-10 minute survey at the end of each day for seven days using Google Forms. Participants will complete the questionnaire the day after to log data for the previous day’s activities.

● Researchers will check-in with participants on the first day of the diary study, and follow-up with participant if they do not complete the daily survey

Interview

● Post-diary interviews will last approximately 30 minutes.

● Interviews will be conducted in person or over the phone.

● Interviews will be audio recorded for transcription and further analysis.

Participant characteristics and desired mix

Characteristic Desired mix

Age

18-30 2-3

31-40 2-3

41+ 1-2

Education

High school/GED 2-3

College 2-3

Graduate+ 1-2

Speaks English

Yes All

Gender




Male 3

Female 3

Participant Details

ID Recruiter Age Range Gender Other Demographics

P01 Lindsay 25-30 Male Police officer

P02 Lindsay 31-40 Female UX Specialist, Portuguese

P03 Lindsay 51-60 Female Homemaker, Volunteer

P04 Jason 31-40 Male Librarian

P05 Jason 31-40 Female Physical Therapist

P06 Leilani 25-30 Female Marketing

P07 Leilani 25-30 Female Marketing

P08 Cam 25-30 Male Apple At-home Advisor

P09 Cam 25-30 Female Teacher

Appendix

Daily Diary Study Log

See the live form on Google Forms: http://goo.gl/forms/wmWz9ZZMKU

To complete this brief questionnaire, please have your mobile device close by so you can refer to the Moment app’s activities from the previous day. This questionnaire should take about 5 minutes to complete. Please note: the questions below pertain to yesterday’s mobile phone activity.

Please answer the following to the best of your knowledge:

1. Name: {single-line text box}

2. This diary study is for the following date: {date-picker}

3. What was your total screen time? {multiple choice}

○ Less than 1 hour

○ At least 1 hour, but less than 2 hours

○ At least 2 hours, but less than 3 hours






○ If more than 5 hours, please enter the total here {single-line text box}

4. What was your total number of pickups? {multiple choice} ○ 0-20 pickups

○ 21-40 pickups

○ 41-60 pickups

○ 61-80 pickups

○ 81-100 pickups

○ If more than 100 pickups, please enter the total here {single-line text box}

5. How long were you using your phone in one sitting? {multiple choice}

○ 1-10 minutes

○ 11-20 minutes

○ 21-30 minutes

○ 31-40 minutes

○ 41-50 minutes

○ 51-60 minutes

○ If more than 60 minutes, please enter the time here {single-line text box}

6. Where are some of the places you travelled while using your phone? {multiple choice text boxes}

○ Home

○ Work

○ School

○ Social space (bar, restaurant, etc.)

○ Gym

○ Errands (grocery store, post office, etc)

○ Other: {single-line text box}

7. Did you look at Moment at least once outside of completing this survey yesterday? {multiple choice}

○ Yes

○ No

8. What are your overall thoughts on your phone usage for the day? {OPTIONAL open comment box} Thanks for completing your survey today!

[SUBMIT]

Moderator’s guide:

https://docs.google.com/document/d/1gVyp_196smEPJcmal6mIY1J90EQYJsgcPVPGPYysMOU/edit#




Consent form:

E-sign: http://goo.gl/forms/pmXREiQ0WE

Print: https://docs.google.com/document/d/1YDSDNsB6CXDsSjzAeQAVBBj8qrLFKpb1empGITtq1ig/edit#

Diary Study Results

Total hours spent on phone per day (in hours)

Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 Week Avg

P1 3-4 1-2 1-2 1-2 1-2 1-2 1-2 1-2

P2 0-1 1-2 1-2 1-2 0-1 1-2 * 0-1

P3 3-4 3-4 1-2 3-4 4-5 2-3 3-4 2-3

P4 1-2 1-2 2-3 2-3 2-3 1-2 2-3 1-2

P5 2-3 1-2 3-4 4-5 5hr 23min 2-3 3-4 2-3

P6 1-2 5hr 9m 6hr 54m 4-5 4-5 4-5 6hr 17m 4-5

P7 2-3 2-3 3-4 2-3 2-3 2-3 2-3 2-3

P8 2-3 3-4 3-4 1-2 4-5 4-5 2-3 2-3

P9 2-3 1-2 4-5 2-3 2-3 2-3 3-4 2-3

Total number of pick-ups per day


P1 41-60 41-60 21-40 61-80 41-60 61-80 61-80 41-60

P2 0-20 21-40 21-40 21-40 0-20 21-40 * 0-20

P3 21-40 41-60 21-40 21-40 21-40 21-40 21-40 21-40

P4 41-60 61-80 61-80 61-80 81-100 61-80 81-100 61-80

P5 41-60 0-20 21-40 21-40 61-80 41-60 41-60 21-40

P6 21-40 61-80 41-60 21-40 61-80 41-60 61-80 41-60

P7 61-80 61-80 61-80 21-40 41-60 41-60 61-80 41-60

P8 41-60 81-100 118 61-80 111 137 61-80 61-80

P9 61-80 41-60 108 41-60 41-60 21-40 61-80 41-60




Longest time spent on phone in one period per day (in minutes)


P1 51-60 21-30 11-20 21-30 11-20 11-20 1-10 11-20

P2 1-10 21-30 11-20 11-20 1-10 1-10 * 1-10

P3 41-50 41-50 31-40 31-40 31-40 11-20 21-30 21-30

P4 1-10 11-20 11-20 21-30 1-10 21-30 31-40 11-20

P5 31-40 31-40 41-50 99 188 41-50 71 41-50

P6 21-30 51-60 96 21-30 21-30 31-40 83 31-40

P7 11-20 11-20 11-20 11-20 21-30 21-30 11-20 11-20

P8 31-40 11-20 11-20 11-20 41-50 11-20 31-40 21-30

P9 21-30 11-20 31-40 11-20 21-30 31-40 31-40 21-30




Appendix C: Journey Map




Appendix D: Personas




Appendix E: Diary Study Spectrums Participant Key




Appendix F: Prototype Original: http://6ft4yt.axshare.com

Revised: http://shlaoj.axshare.com/

Recommendations for Changes

Change to App Supporting Data Should this be implemented now?

Change menu item from “Screen Time” to:

● Metrics ● History ● Data ● Usage

History ● Time on

Screen

Cog WT: “Screentime is at the top and in the bottom menu. I clicked the wrong one.” “because the icon of "ScreenTime" in the bottom of the page doesn't represent past logs.” “So your logo is called "ScreenTime," and the actual task activity is also "ScreenTime." So when you ask me to click on "ScreenTime," I clicked on the logo because it was the first thing I could see.” “I feel like using another term is appropriate on the t2 home page. maybe something like usage or history instead of screentime.” “The action would be more straightforward if it was labeled more clearly” Cog D2L: "ScreenTime" […] It's buried at the right corner of the bottom nav so when you asked me to click on it I went straight for the ScreenTime logo which is at the beginning of the Z-pattern instead of at the end. Can't tell you how many times I clicked on that logo! :)

Y Y

Show username after login

Cog WT: “Because the name of the user doesn't show up after the registration.” N N N N

Improve login validation form

Cog WT: “The prototype doesn't alert you to a password error. So you can type in two different passwords that don't match. You'll probably going to build that out in further iteration to correct that.” “I noticed the password form field is the only which easily replaced the example text in the field. When entering the username and email, the user must highlight and delete the text.” “Currently there are no error messages for incorrect information input. I would add those for usability robustness.” Cog D2L: You do not have to enter the email address with any extensions ex. @gmail.com. Cog D2L: I would add a confirmation dialog box to confirm to the user that the information was excepted. Cog D2L: there is no link or button for users who already have created their

N N N N




accounts to sign in.

Provide an improved privacy message for the autotrack screen and sharing data with friends/global

Cog WT: “The only challenge is perhaps the fact that the app tracks your usage and there is no mention of what the privacy would be, thus making the decision to turn on the tracking and complete the task may cause them to leave the task unfinished.” “Wording on explanation of autotracking might want to be looked at again” “Add a little bit of information regarding the privacy of the tracking.” Cog D2L: I was concerned about the other tap "Your Friends". I didn't feel comfortable sharing with my friends how much time I spend on the screen. Maybe because I didn't see any settings page where I can turn off this feature or "running in background" feature.

N N N N

Combine the username and email field

AP03: “Sign up screen looks busy - would combine username and email fields. “ N N N N

Make “This week” and “All time” on home screen clickable into the Timeline Usage screen.

AP04: “Tried selecting the "This Week" tab and then the "All Time" tab. Was convinced that this was where to go” AP 03 “"I would normally select the buttons at the top of the page" (instead of selecting a menu item at the bottom) “ AP 01 “Tried to view the "All Time" tab, was looking for a function to select a date. “ Cog D2L: I also found clicking on the "ScreenTime," at the bottom to be an extra step when getting Today usage data. Why not just have me click on the "55 Pickup," selection on the screen for that breakdown.

Y Y Y Y

On homepage, make “Pickups” and “Hours” clickable into the days timeline.

Cog WT: “It just seems easier to click on the "55 Pickups Today," and get today's data. Why have to go down to the bottom, click on ScreenTime see a list of days and then click "Today," when "Today's data," is the first choice on the screen? I know this is a test of a task but it seems a bit counterintuitive.”

N N N N

Redesign the Goals interface (new goal > customize goal > confirm)... exact changes TBD Potentially include on-screen tutorial for getting started with the app or “getting started” messaging on dashboard view.

Cog D2L: The last task "Goal Setting," the interface was a bit confusing. Not sure why those were split up into different screens. It seemed extra work since Setting a goal, Giving a Goal Deadline and Saving a Goal are all part of the same activity. Cog D2L: I wasn't sure my Goal was saved (outside of the successful completion of the task notification). AP04 Was unsure if the task was complete because there was no save option. He wasn't sure if it had been confirmed, so he clicked the checkbox next to his new goal.

Y Y YY




Cog D2L: I found the use of a plus button to enter a goal to be confusing.

Redesign the Goals flow so that it’s better integrated with the usage metrics since participants do not understand the connection between the two… exact changes TBD

Cog D2L: It would seem that if the setting of the goal's purpose is to shift time - from app time to goal time - that there might be time on goal to set as well and not just a deadline. So if the app says you have 55 Pickups to get 3 hours...you might want to show a way to "shift time," from apps to the goal so the person completes the goal by the deadline. For example: Let's say you have a goal of completing your paper in two days and you need four hours to work on it. You might show the user how to cut time from each app to get that four hours. Cog D2L: it seems like there are a lot of things going on within the app. It's hard to pinpoint what the exact purpose is. I'm conflicted between a calendar/reminder app and a screen monitoring one. Cog D2L: The functionality worked fine but, it was presented in a way that made it seem out of scope. I thought this was to reduce screen time mobily. That's how you knew how many pick up times there were. Right? But, then your app mentioned midterm notes (not mobily viewed right?) then I got confused. Cog D2L: Set a goal, I am not sure what this task has to do with the app. Does it give the user a reminder when of their goals when they reach a certain number of pickups? If that’s so, I think it is a cool idea. [...] When I added the information for the goal I expected to see some changes on the goal item in the Goals screen, like changing the color or adding a timer icon to tell the user that their information is saved and to make it easier for them to know what kinds of goals they have from the list. AP02: “It's very clear when you click on goals and you say add your goals, yea I wanna make a new goal. But then I have to type my own goal in there. And what I thought would happen was I would click plus and then a list of my apps would come up so I could select facebook and say "I wanna use facebook 2 hours less this week" or something, something I feel like I would get notifications or it would affect my data somehow or the app would be able to tell me if I achieved my goal or not versus something like, "finish your book", well the app doesn't know if I finished my book or not. So it seems kinda outside the scope or like its not all working together.”

Y Y Y Y

Make the labeling of “exclude apps” feature consistent

Label inconsistency: In the settings = “exclude apps” In the menu = “limits” On the usage screen after you click one of the circles = “filter apps that you don’t want to be tracked” Cog D2L: For "limits" I think it would be clearer to use the same term you used on other page, which is "Filter". Cog D2L: reduce the cognitive load of having to go through certain steps to accomplish some of the intended tasks, especially filtering, which could be combined with limits and checking other dates screen times.

Y to making the label consistent (no to changing how you can access this feature) Y , agree with label consistency changes, and do not change the method of access). Y

Provide easier access Cog D2L: one thing I found challenging which is the clicking the colored circles Y Y Y




to the metrics that provide the app usage breakdown (Drill down concept)

to see the applications I used. I would have missed that if I didn't see it in the instructions. This task (3A) had a 38% success rate with our assessment test participants Cog Walk: “Clarity of the proper selection is a challenge. Looking at the home screen, I would think to click on the three buttons near the top to get to the 23rd before going to the bottom menu.” “Adding the ability to change to the previous day on the same screen would be helpful to reduce cognitive load, to reduce the need to figure out where the days are located” AP02: The first screen where you see like the days of the week are okay, although I would expect that clicking on Weekly View to get to that. This thing with the yellow and red dots there' all in a vertical line. That doesn't show me, that seems like a weird visualization for how the day would go.

Remove boxes on right hand side of “Usage” screen

Cog Walk: The check box in the right side of the previous days are unnecessary since the user can click on the day and get to the next page.

Y Y Y Y




Appendix G: Cognitive Walkthrough

Test Plan

Objectives The purpose of this study is to leverage usability experts to evaluate a set of five tasks within our initial application prototype. Our goal is to understand the learnability of our initial application prototype. In addition, we’d like to identify possible usability issues in our design. The results of the study will be used to provide a recommendation for a first revision of the prototype. Research Questions

● Do the actions of the application flow appropriately? ● What tasks would a user of the application consider challenging? ● Does the user have all the tools they need to be successful in the task? ● Will the customer be confused by language or actions in the system? ● What aspects of usability in the prototype are strong? Which aspects need improvement?

Methodology In order to explore the answers to the questions above, we will consult usability experts to complete a cognitive walkthrough of an initial prototype design. These experts will be asked to complete the following tasks (for a detailed, step-by-step list, please see the Appendix). Procedure

Included in the instructions planned for D2L, participants will be asked to go through the following process: 1. Open the prototype and the evaluation form using the links posted in D2L. 2. Review the purpose and instructions for completing Cognitive Walkthrough (See Appendix). 3. For each task:

a. Answer each question provided for each task in the evaluation form: i. Please provide a rating for this task (1 = Very Easy, 5 = Very Difficult) using a likert

scale. ii. Will the user notice that the correct action is available? iii. Will the user associate the correct action with the effect to be achieved? iv. Will the user try to achieve the right effect? v. If the correct action is performed, will the user see that progress is being made toward

solution of the task? vi. What challenges would a user experience with this task? vii. What other comments or suggestions do you have pertaining to this task?

b. (Optional) Upload images or diagrams of suggestions for improvement.




4. Submit the form and ask any questions using the D2L thread or a researcher’s email.

Test Logistics ● The instructions will be posted to D2L using the class submission template (See Appendix). ● The test will open on Monday, February 22, 2016 and close on Saturday, February 27, 2016 or when

we have received six responses (whichever happens first). ● The low-fidelity, interactive prototype will be made available through AxShare online software:

http://6ft4yt.axshare.com/ ● We will collect the responses of usability experts using an evaluation form via Google Forms. ● Experts will be able to upload documents that allow them to upload to our groups Google Drive

folder. Experts will not be able to see others responses or delete/modify the files. ● Experts will not be compensated for their time by our group. Completion of our cognitive

walkthrough will contribute towards a class participation grade.

Appendix

Cognitive Walkthrough Submission Form View the evaluation form using Google Forms

D2L Entry Name: ScreenTime Cognitive Walkthrough

Team: Team 2

Activity Open Dates: Monday, February 22, 2016 - Saturday, February 27, 2016

Number of Participants Needed: 6 participants (maximum). Study will close after 6 responses are submitted.

Activity Objective: The purpose of this study is to understand the learnability of our initial application prototype called ScreenTime. The purpose of ScreenTime is to help users better understand their phone usage habits and how to alter their behavior to focus more on achieving their goals towards a more balanced lifestyle. In addition, we’d like to identify possible usability issues in our design. Please note that this is a LOW-fidelity prototype, and that the results of the study will be used to provide a recommendation for a first revision of the prototype.

Estimated Activity Time: About 30 minutes

Activity Instructions:

1. Open up the prototype on a tablet or desktop using this link: http://6ft4yt.axshare.com/




2. In a separate window or tab, open up the evaluation form: http://goo.gl/forms/8TCYhap3o1 3. In the evaluation form, you will be presented with 5 tasks to complete. Please use the step by step

instructions located on the evaluation form to complete each task. 4. At the conclusion of each task, answer each set of questions for each particular task. 5. If desired, please upload any visual examples of ways to improve any particular task here:

http://tinyurl.com/jxhed7f. 6. Submit the form.

For any issues or questions, please post to this D2L thread or reach out to [email protected].

Individual Task Descriptions Please complete each task according to the instructions below. You may open the prototype and the Google Form on your desktop or you can interact with the prototype on your mobile device (preferred method).

1. Task 1: Sign-up for the ScreenTime app. a. Navigate to http://6ft4yt.axshare.com/ and select “Task 1: Sign-up for the ScreenTime app”. b. Create a unique username in the first textbox. c. Enter in your email. d. Create a password. e. Confirm the password you just created in the next textbox. f. Select the “Sign-up” button. g. Start tracking your usage. h. Select “Finish”.

Task complete: Select “Go to Prototype Landing Page” button for Task 2. Complete the questionnaire for Task 1 using the Google Form.

2. Task 2: Check your usage statistics for the previous day. a. Return to the prototype landing page and select “Task 2: Check your usage statistics for the

previous day”. b. Select the “ScreenTime” option from the menu. c. Select the option for “February 23rd” and view the usage data.


3. Task 3 - View today’s specific app usage and exclude apps that you don’t want to be tracked. a. Return to the prototype landing page and select “Task 3: View today’s specific app usage and

exclude apps that you don’t want to be tracked”. b. View pick-ups and screen time for today. c. Select the “ScreenTime” option from the menu. d. Select the option for “Today” and view the usage data. e. Press one of the first two colored circles and view application usage. f. Select “Filter apps that you don’t want to be tracked”. g. Select one or more applications to filter.




h. Select “Save Filters”.


4. Task 4 - Set a goal. a. Return to the prototype landing page and select “Task 4: Set a goal”. b. Select the “Goals” icon from the menu below. c. Read the help text and select “Got it!” d. Select the plus icon to add a goal. e. Type in a goal you’d like to complete in the next week in the textbox. f. Press the plus icon to complete adding the goal. g. Select “Goal info”. h. Activate a goal deadline. i. Enter in a due date for two days from now. j. Enter in the messaging you’d like to include for the reminder. k. Select “Save Goal Details”.


5. Task 5 - Compare your usage statistics to others. a. Return to the prototype landing page and select “Task 5: Compare your usage statistics to

others”. b. Select the “Other People” icon in the menu. c. View General and then view Your Friends to view their usage data.

Task complete. You may exit the prototype at this time. Complete the questionnaire for Task 5 using the Google Form.

Once you complete the questionnaire for Task 5, please submit the form. Thank you for your participation!

Cognitive Walkthrough Results

Feedback from Experts for Task 2 - “Check your usage statistics for the previous day”

E1 E2 E3 E4 E5 E6


Will user notice the correct action is available? N N Y N N N


N N Y N Y Y





If the correct action is performed, will the user see that progress is being made toward

solution of the task?

Y Y Y Y Y N

Feedback from Experts for Task 3 - “View today’s specific app usage and exclude apps that you don’t want to be tracked”

E1 E2 E3 E4 E5 E6


Will user notice the correct action is available? N Y Y N Y Y


Y N Y N Y Y

Will the user try to achieve the right effect? Y Y Y N Y Y



Y Y Y Y Y Y

Feedback from Experts for Task 4 - “Set a goal”

E1 E2 E3 E4 E5 E6


Will user notice the correct action is available? N Y Y Y Y Y


N Y Y Y Y Y

Will the user try to achieve the right effect? Y N Y Y Y Y



Y N Y Y N Y

Feedback from Experts for Task 5 - “Compare your usage statistics”

E1 E2 E3 E4 E5 E6





Will user notice the correct action is available? Y Y Y Y Y Y


Y Y Y Y Y Y




Y Y Y Y Y Y




Appendix H: Assessment Test

Test Plan

Objectives Our study focuses on providing people with an opportunity to understand the relationship between phone usage and productivity in order to improve their lifestyles. The first activity in our study, the competitive evaluation, revealed that existing apps provide phone usage data to users, but they draw little to no connection between phone usage and improving lifestyles or behavior. Our second and third activities, the diary study and post-study interviews, revealed that users assumed their phone usage was high, but were unsure whether or not their data was “normal” since there was no benchmark for them to reference.

We have taken these findings and created a low-fidelity prototype called ScreenTime. For our final activity, the observation study, we want to assess the satisfaction and effectiveness of two features that we have introduced that are unique to our prototype: setting up goals and offering comparative data so that users can see how their data stacks up with others. The results of the study will be used to provide a recommendation for a first revision of the prototype.

Research Questions

Research Question Measure Metric

Assessing effectiveness

Are users able to complete basic tasks, such as creating an account, setup, checking their own stats

Quantitative Success Partial Success Fail

Are users able to choose an existing goal and create their own?


Are users able to compare their stats against global and their friends’ stats?


Assessing satisfaction




Do users enjoy using the app? Qualitative Think aloud commentary Facial expressions Post-task commentary

Do users like being able to choose from and set up their own goals?

Qualitative Think aloud commentary Facial expressions Post-task commentary

Do users like being able to compare their stats against other stats?

Qualitative Think aloud commentary Facial expressions Post-task commentary

What tasks did the user find challenging? Qualitative Think aloud commentary Facial expressions Post-task commentary

Measurement Criteria

SUCCESS = 2 points

● Participant was able to complete the task as defined by the success criteria. ● Participant only needed to ask one or less question in order to complete the task. ● Participant did not need any guidance from the moderator in order to complete the task. ● Participant completed the task within 3 minutes.

PARTIAL SUCCESS = 1 point

● Participant was able to complete the task as defined by the success criteria. ● Participant needed to ask 1-2 questions in order to complete the task. ● Participant needed minimal guidance from the moderator in order to complete the task (1 or less

assists). ● Participant completed the task within 5 minutes

FAILURE = 0 points

● Participant did not complete the task as defined by the success criteria. ● Participant needed to ask 3 or more questions in order to complete the task. ● Participant needed a lot of guidance from the moderator in order to complete the task (2 or more

assists). ● Participant completed the task in more than 5 minutes.




Methodology To answer these questions, we will conduct an observation study with ScreenTime, our low-fidelity prototype using Axure. Team members will observe four participants completing a set of tasks in ScreenTime on the moderator’s phone. Each participant will go through the same set of tasks indicated in the procedure section. Participants will be encouraged to “think aloud” as they’re progressing through the tasks so that the moderators can collect feedback. The moderators will also ask follow up questions to gauge overall satisfaction and/or challenges with the app (see Moderator’s Guide in the Appendix).

Each team member is responsible for recruiting and observing one participant. Participants must be smartphone owners who are least 18 years old. While they may be familiar with apps similar to Screentime, they may not be familiar or have participated in our interview or diary studies. We will ask a set of pre-screening questions prior to the test session. Participants will not be compensated.

Participant characteristics and desired mix

Characteristic Desired mix

Age

18-30 1 or 2

31-40 1 or 2

41+ 1

Education

High school/GED 0 or 1

College 1 or 2

Graduate+ 1 or 2

Speaks English

Yes All

Gender

Male 2




Female 2

Participant Details

ID Recruiter Age Range Gender Email Other

AP01 Lindsay 18-30 Female [email protected]

Bachelor’s degree

AP02 Jason 18-30 Female [email protected] Graduate

AP03 Cam 18-30 Male [email protected] Graduate

AP04 Leilani 31-40 Male [email protected]

Some college

Procedure 1. Introduction 2. Project screening and details 3. Review and sign consent form 4. Setup: place the smartphone on desk/table. Turn on camera (if applicable)

a. Note: Moderator to save prototype to home screen 5. Tasks

a. Task 1: Sign-up for the ScreenTime app. b. Task 2: Check your usage statistics for the previous day. c. Task 3: View today’s specific app usage and exclude apps that you don’t want to be tracked. d. Task 4: Set a goal. e. Task 5: Compare your usage statistics to others.

6. Wrap-up questions

Test Logistics ● Observations will occur from Monday, February 22 through Saturday, February 27, 2016. ● Participants will use the moderator’s phone for testing. ● Moderator’s role:




○ Each team member will serve as a moderator, and there will be only one moderator in each test session. The moderator will be responsible for the entire session, from greeting the participant at the beginning to documenting the notes after the participant has left.

○ During the session, the moderator will read directly from the script. ○ The moderator will sit next to the participant. ○ The moderator will take notes during each session. Quantitative data will be transferred into a

Google spreadsheet. Qualitative data will be transferred into a Trello board. All participant data will be anonymized.

● We will audio record every test session ● The low-fidelity, interactive prototype will be made available through AxShare online software.

Appendix

Moderator Guide

INTRODUCTION

My name is [insert name], and I would like to thank you for agreeing to participate in this study. As you may know, I am working towards my master’s degree in Human-Computer Interaction at DePaul University. This study is being conducted in partnership with three other teammates in one of my courses. Before we begin, I’d like to explain the purpose of this study and address any questions or concerns you may have.

The purpose of this study is to get your opinions on mobile apps that tell you how much time you’ve spent on your phone every day. We are trying to find out whether or not this information is meaningful to you, do you like seeing this type of information, what types of information would you like to see, etc.

I will also give you a set of tasks to complete. As you’re working on each task, I’m going to encourage you to “think aloud.” Meaning, talk through your actions as you’re doing them. This may seem a little awkward at first, but hearing your thoughts is really beneficial to our team. So, for example, if this was a retail app and I asked you to review the items that you plan to purchase, you may tell me that you’re looking for a shopping cart link or button.

Also, the app that you will be reviewing today is a prototype, meaning that you are previewing a very early version of the app. It’s not fully designed yet, nor is it fully built out. There may be times when you try to tap something that seems like it’s broken or doesn’t do anything. That’s totally fine and expected. In these times, thinking aloud will come in handy because I’ll be able to know what you would have expected if this was a fully functioning app.

As you’re going through the session, you will see me jotting down notes. I am also recording this session so that I can review with our team. I should also mention that this is an evaluation of the app, not an evaluation on you. This not a test, and there are no right or wrong answers. Overall, we are get your honest feedback,




regardless if it’s good or bad. Our goal is to see whether the app has been designed to match what comes natural to you.

The entire session will take about 30 minutes.

Before we begin, do you have any questions?

CONSENT FORM

With everything I mentioned above, we do ask that participants sign a brief consent form just to keep it on file. Can you please review this form and sign it? [Hard copy: https://drive.google.com/open?id=1H_vEScyxX94cTAGoxgu1gKTkYu1v0HHEJOO9ZfL1E1s OR e-sign: http://goo.gl/forms/QNsakx4S6X.

TASKS

The app that we’re going to review today is called ScreenTime, which tracks your daily phone use, such as how many times you unlocked it, total number of hours, and which apps you spend the most time on.

Have you heard of an app like this? How you used one? [note participant’s response]

I’m going to give you a few tasks to complete in ScreenTime. Remember to talk through your actions as you complete each task; no action you take is too small nor is any action incorrect.

[Give moderator’s phone to participant with ScreenTime AxShare icon present on the home screen. Link: http://n3hyl3.axshare.com/#c=2]

[moderator should not provide help in completing any of these tasks unless participant “gives up”.]

Task 1A: Sign-up for the ScreenTime app.

Can you please launch ScreenTime? [success if participant taps ScreenTime AxShare icon]

Could you show me how you would create an account with your email address and complete setup? [success if participant chooses sign up, enters information, and toggles on autotracking, clicks Finish]

That completes the first task. On a scale from 1 to 5, could you tell me how difficult this task was, with 1 being very easy and 5 being very difficult?

Task 1B: Interpreting the Metrics

[User should be on home page with the 55 pickups and 3 hour screentime from previous task.]

Could you tell me what you see on this screen?

What do you think these numbers mean?

Task 2: Check your usage statistics for the previous day.




Okay, let’s say that you wanted to see how much time you spent on your phone a few days ago. Could you tell me how many minutes you spent on your phone on February 23? [success if participant goes to screentime icon, and says that they spent 204 minutes on Feb 23]

That completes this task. On a scale from 1 to 5, could you tell me how difficult this task was, with 1 being very easy and 5 being very difficult?

Task 3A: View today’s specific app usage

As I mentioned before, ScreenTime will tell you which apps you spend the most time on. Where do you think you would find this information? [success if participant goes to Timeline, clicks circle and lands on app usage screen]


Task 3B: Exclude apps that you don’t want to be tracked.

Now, let’s say that you don’t want to include Skype since you use it to communicate with your coworkers during the workday, so it really shouldn’t count. What would you do? [success if participant checks Skype from list, taps Save Filter]


Task 4: Set a goal.

Now on to the fourth task. Let’s say that you’ve realized that you’re spending too much time on your phone, and you really need to finish reading your book by next week. Could you show me what you would do? [success if participants goes to Goal, creates new goal, sets deadline, Saves Goal]


Task 5: Compare your usage statistics to others.

Now we’ll do the final task. Could you tell me how many times your friend Mel Bryer has picked up the phone today? [success if participant goes to Other People, Your Friends, and says that Mel Bryer had 60 total pickups]

That completes the fifth task. On a scale from 1 to 5, could you tell me how difficult this task was, with 1 being very easy and 5 being very difficult?

FOLLOW UP QUESTIONS

Now that we’re done with the tasks, I’ll just ask a few questions.

1. What did you like about the app?




2. What did you dislike about the app? 3. What did you think about setting goals? 4. What did you think about comparing your stats? 5. Are there things that you would change about the app?

That’s all of the questions that I have. Do you have any questions for me?

THANK YOU

Thank you so much for taking the time to complete the diary study and answering a few questions for me today. If you can think of anything you’d like to add after this, please don’t hesitate to reach out! Thank you.

Documents

Using Metrics from Mobile Technology to Improve Self- Awareness …media.virbcdn.com/files/77/9e0510b2f513fe72-FinalReport... · 2016-05-07 · HCI 594 Final Project Report Team: