Embed Size (px)
Citation preview
What is the best way to learn about the experience that people are having with your product?
3
Interviews don’t capture the range of ways people do things or the rich, in-the-moment details that people quickly forget.
Digital diary studies allow you to: • Get instant feedback as people naturally use a
product or service • Ask them follow-up questions throughout the study • Access everything through a digital platform
5
Are You Wearing One?
Fitness Tracker Case StudyWe recruited 40 people who use fitness trackers to participate in several digital diary studies over 8 months.
The first diary study had them report on 5 things they do with their tracker. With every entry, they submitted a short video and answered a few questions. Over 170 entries were collected.
We did 4 other diary studies with them over the course of 8 months to better understand how their fitness tracker usage evolved and the role it plays in their overall health goals.
6
7
Erica, 33
“It’s 10:53pm, and I’m only at 7388 steps….we’re about to go take the dog for a walk. Got to make sure we get to our 10,000 steps before the end of the night.”
What This Presentation Will CoverWhat a Diary Study is
The Benefits of Using Diary Studies
How Diary Studies Compare to Other Research Methods
Traditional Diary Studies vs. Digital Diary Studies
What the Participant Experience Is Like
How to Create a Digital Diary Study
Findings From Our Fitness Tracker Diary Studies
8
What is a Diary Study?
A Diary Study is a Cultural ProbeCultural probes use a collection of tools, artifacts and tasks intended to provoke the user to think about their environment in new ways….With minimal intrusion, researchers can glean insights into participants’ environments that can help to identify problem statements and uncover new opportunities.
10
Adam Starkman & Jennifer Chow http://designresearchtechniques.com/casestudies/cultural-probes/
Diary Studies Have 4 Key ElementsTheme-based: the subject and questions are suggested by the researcher.
Self-reported: the actual reporting is done by the participant, using whatever platform/prompts were provided.
In-the-moment: the participant is asked to do the reporting immediately after engaging in the activity, in order to capture as much detail as possible.
Longitudinal: the goal is to capture a range of uses, rather than a singular experience. This can include how the user’s experience evolves over time.
11
Traditional Diary StudyMozilla used paper diary studies in 2012 to learn how participants were using bookmarks in Firefox.
They asked 10 people to record their experiences with bookmarks over 3 days. Each day had a different set of questions.
12
Brian Groudan, Mozilla research, 2012 / https://blog.mozilla.org/ux/page/24/
Traditional Diary StudyPeople came in for an in-person interview afterwards, where they discussed their entries and sorted them by theme.
Guiding design principles were culled from the analysis that were the basis for design better bookmarking tools.
13
Why Use Diary Studies?
Looking for the DetailsThe goal of research is to understand not just what people do (and want and believe), but why.
This is complex and requires a deep understanding of people’s behavior.
Knowing details about their behavior is necessary to get to that understanding.
15
However people often have a hard time remembering those details when not engaged in the activity.
This is especially true for habitual behaviors.
Habits are formed when our brains:
Recognize that we repeatedly do the same sequence of actions (like checking email) after being triggered by a specific cue (such as picking up our phones).
The brain “chunks” that sequence into one automatic routine.
This allows our brain to save mental energy for more complex tasks.
Barriers to Collecting Details
16The Power of Habit, Duhigg
A consequence of having a sequence of behaviors “chunked” together is that details of the habit become accessible only after the cue has been activated.
So outside the actual ‘doing’ of the task we have difficulty remembering the details.
This is why gathering information in context collects such rich details.
Barriers to Collecting Details
17The Power of Habit, Duhigg
18
The best way to collect these rich details are either observation (contextual interviews) or in-the-moment reporting (diary studies).
How Diary Studies Compare to Other Methods
20
Quantitative methods only explains what people do while qualitative methods offer insight into why people do them.
Focus Groups
Diary Studies
Usability Testing
Surveys
Big Data
Contextual Interviews
why people do it
what people do
Quantitative (conscious & logical)
Qualitative(subconscious &
emotional)
21
Once you understand why, your insights can move from evaluative (improvement) to generative (new opportunities).
Focus Groups
Diary Studies
Usability Testing
Surveys
Big Data
Contextual Interviews
evaluative
Data
Stories
generative
22
Yesterday’s session about Storytelling by Mary Wharmby discussed this.
23
Karen, 26,grad student
“I got drunk with my friend and convinced her to download Snapchat on the contingency that I send her a [Snapchat] everyday. I use my [Jawbone Up] to remind me at 7pm to send her a picture….it vibrates on my wrist.”
How to Use Diary Studies with Other Research Methods
Insight Study ProcessAn Insight Study is a series of 2-3 focused research cycles built around a specific question. Multiple research methods are used in order to capture different types of insights.
Each cycle concludes with an Insight Review that shares the findings with the client team.
The benefit for the client is that they:
• Deliver real-time findings that can be acted upon right away
• Offer an opportunity to refine the next phase of research based on the learnings.
25
26
Insight Study Methods: Contextual Interviews allow us the deepest insights by observing people’s usage firsthand.
Diary Studies collect in-the-moment behaviors around a variety of issues throughout the project.
Remote Interviews fill in the gaps by probing deeper into the diary study entries and related areas.
The Benefits of Using Mixed MethodsUsing mixed methods allow us to validate our findings and take advantage of the unique strengths that the various research methods offer.
• Diary studies let us collect actual behaviors around a range of issues or track ongoing usage patterns.
• Contextual interviews give us the deepest insights by letting us observe their natural environment directly.
• Remote interviews give us access to people who are geographically distributed and allow more frequent opportunities to engage with them.
27
What is a Digital Diary Study?
29
Getting people to actually do the reporting is a major obstacle for traditional diary studies.
http://kotaku.com/hong-kong-residents-supposedly-cant-live-without-their-476758731
30
Smartphones are ideal for digital diary studies since people use them in short spurts throughout the day.
http://kotaku.com/hong-kong-residents-supposedly-cant-live-without-their-476758731
31
Digital diary studies can be as simple as text or email prompts that people reply to.
Digital diary study platforms
32
Name Platform Cost
24Tru Web/app diary study high
CrowdTap Web/app diary study high
UX 360 Web diary study high
Dscout App diary study medium
Dedoose Web diary study low
Tumblr private blog low
Typeform online survey low
33
$595 for recruiting (or you can bring in your own participants)
$295 for each study
34
Dscout has great guides and online support to help get you started.
The Participant Experience
36
This view is only if recruiting through dscout.
Participant reviews opportunities on app
Apply for a study Fill out screener
37
Once invited to a study, this is what all participants see.
Email invite to mission Q1: Begin with media prompt (photo, video or text)
Inside the app
38
Studies can have up to 10 questions each, and can be any mix of multiple choice, rating, or open-ended questions.
How to Conduct a Digital Diary Study
The 3 Phases of a Diary Studies
40
The goal: a concise, focused study with the right participants.
The goal: high participation and high quality data.
The goal: to turn data into relevant insights.
• Clarify your research question
• Refine and pretest your study
• Minimize the work for your participants
• Recruit the right participants
Clarify Your Research Question Our Initial Questions:
How are activity trackers used by people attempting to make concrete improvements to their fitness habits?
Can we discern patterns around when activity trackers are more or less effective at helping people make those changes?
43
44
Make a list of possible questions. You will have too many. Edit and sort them by theme.
45
Define the goals for each study. This will help maintain focus as you refine the questions.
46
Diary Studies are basically surveys with a twist.
47
That means answer options, which require a lot of refining. The more time spent building the study, the better the data will be.
Make it Easy for your Participant to SucceedA guiding principle as you build your study should be to minimize the work your participant has to do.
48
49
Consider how often you ask them to report. It should be enough to collect useful data, but not enough to cause fatigue.
50
Pairing themed multiple-choice questions with open-ended ones creates a logical flow for people and answers both “what” and “why”.
51
Another benefit of multiple-choice is you can quickly see trends as the data comes in.
52
Open-ended questions provide the richest data but are more work for both for you and the participant. Use them sparingly.
I think it should send reminders on when to be more active when I'm lagging in my goal such as vibrating.
I do wish that other activities could be tracked with the Fitbit without having to input it myself.
Maybe if I could get some sort of notification on my phone if a specific friend or group of friends had passed me.
53
Pretest by sitting down with someone as they go through every step. Look for awkward or unclear phrasing.
54
Media options: video adds a lot, but also adds more work for both you and the participants.
Take a picture that symbolizes the role your tracker plays in your health/fitness goals
Tell us what ideal health would look like for you.
55
Some questions are better suited for a photo, which makes it easier to compare (and share) responses.
A long road ahead. Before last year I never thought I could be THAT mom who was fit & healthy.
My tracker is a set of tools I'm using to build a healthier life for myself.
56
When capturing real experiences or complex ideas, video is better.
Karen, 26,grad student
“Being able to walk upstairs without being out of breath. Also, be at the pick of my thinking and not feeling tired and drained because of ill health. And I think that my tracker helps me with that especially with sleep.”
58
Diary studies generate a lot of data. It is good to start small.
10 x 10 x 10 =questions participants entries
data points
1000
59
Recruiting options include customers, friends, Craigslist, social media and dscout.
60
If recruiting through dscout, you’ll only pay incentives for participants that finish. dscout recommends paying $5-10 per entry.
61
We looked for participants who have a fitness tracker and were regularly using it as part of a fitness or weight loss goal.
62
Typical dropout rate is 15-25%, so over-recruit.
40
34
63
You can view each applicant’s responses inside the screener survey and rate them based on how good they are.
64
It’s easy to add participants if you do your own recruiting. They get an email link for downloading the app.
• Review & make notes from your data as it’s coming in
• Be persistent with participants
• Ask them questions for clarity
• Do post-study interviews
67
Watch the data as it comes in. This is the fun part.
What is your favorite thing about your fitness tracker?
Kayla, 27,teacher
“My favorite thing is that by a single click of a button, the display shows me the time, date, exact steps I've taken, my heart rate, steps climbed and calories burned as opposed to having to find my phone, unlock, open the app...it's all right there, easy to read. I like having it all right there on the display. .
68
Entries are reviewed individually.
69
Take notes of interesting patterns and questions to ask participants as you go through it.
70
Moment of truth: are your questions holding up? If not, can you fix it? Either way, learn for next time.
72
Stay in touch with your participants. Send them reminders if they haven’t submitted an entry. Be friendly and persistant.
73
Ask them questions if you want more clarity about their answers. Be appreciative and interested in what they have to say.
74
Do follow-up interviews with the most engaged participants. You already have rapport and can dig deeper into their actual behaviors.
• Start by answering your initial questions
• Use the built-in analysis tools
• Identify patterns and/or exceptions
• Share your findings with real examples
76
The good news is that by simply reviewing and making notes on the data, you have already started on the analysis.
What Further Analysis Should be Done?That depends on many variables, such as:
• What your initial question/s were
• How surprised you are by the data that has come in
• How it fits into other research methods
• Your final deliverables
77
78
The internal tools dscout offers are helpful. Tagging can quickly reveal patterns.
79
Multiple choice questions can be filtered. Here we wanted to review the entries in which people were not as satisfied with their tracker.
For Those Who Rated an Experience with Their Tracker as “OK” or Lower: • 30% were unhappy with some aspect of the band
(too bulky, not water resistant or something had broken)
• 25% were complaining about the charger or battery
• 15% had issues logging their food (either through a secondary app or via their main device)
• The remaining 30% were a random mix of issues
80
Once the responses were filtered, it was quick to manually categorize what remained.
81
Transcribing video is time-consuming, but makes the data much more accessible. Tagging is an ideal way to remember interesting video.
82
Downloading the data into a spreadsheet that can be coded, refined, sorted and easily reviewed is our preferred method if we intend to dive deep.
83
If the original answer options were not accurate enough, you’ll have to throw data away or recode it yourself; here we recoded it.
Fitness Tracker High-Level Findings
85
Why did you get your first fitness tracker?
Larry, 31
“I got a fitness tracker because was about 300 lbs, on a diet, exercising, wasn't losing weight. I knew there was something missing. I wanted to learn how many calories I was burning so could know how little or much I should be eating. That was the missing link. I have lost 150 lbs, and the only way did it was because I started using Jawbone Up.”
People Love Trackers Because of What They Learn About ThemselvesFitness trackers teach people things about their eating and exercise patterns they’ve never known before.
They had made guesses, but they had guessed wrong.
People usually overestimated the exercise they were getting and underestimated how much they were eating.
Knowing the actual numbers gives people control so they can get to the outcome they want.
86
87
What do you like least about your tracker?
Amy, 38
“Now for least favorite. Even though I loved a lot about all the trackers: Omron and Fitbit, the one thing I didn't like was that after a while I got bored. There's only so much competing you can do, so many steps you can take, but they didn't do anything with the data. It was great when I reached my 10,000 steps for the day, but then what? It doesn't help me do anything more with it.”
Once People Learn their Patterns, they get Bored (and/or Frustrated)Depending on how motivated people are to make changes to their health behavior, they usually go through a slump after they learn their patterns.
Changing exercise and eating pattens require significant changes to one’s lifestyle, and knowing one’s patterns is only the first step.
People want more help getting to their goals.
88
89
What does ideal health look like to you?
Mary, 42
“My ideal health would be to feel better, not drag, and eating better foods. It would help me feel motivated to follow my exercises I need a tracker that would also [suggest what] types of foods to eat. It knows that I am running and this is the food you should eat. That would be so wonderful! That would help me a lot even more with my runs, my exercise, with getting in shape.
Trackers Only Address a Fraction of People’s Health GoalsWhen asked what their ideal health looks like, people talked about both complex ideas (such as mood, energy level, endurance and eating habits), and hard numbers (like weight and body fat).
Trackers only help with some of the aspects of ideal health.
People want their trackers to know their goals, learn their patterns and offer suggestions to help them achieve their goals.
90
Digital Diary Study Key Takeaways
Main Benefits of Using Diary Studies• To collect rich stories about how and why people do things
• Get intimate glimpses of how a product fits into people’s lives
• Allows for tracking ongoing use of a product
• Cost-effective way to collect contextual insights
92
Tips for Building Your Own Diary StudyBefore you can use it, you may have to sell it. Here are a few things to keep in mind:
• It’s an extremely flexible tool; make it work for your needs
• Start small, let it prove it’s value to your organization and yourself
• Evangelize the insights and stories you collect to build more support.
• You become an expert by doing. So dive in!
93
What Will You Do with Diary Studies?
For a 4-min video about diary studies: www.gotoresearch.com/speaking/sara/
Questions about diary studies? Ask!
@ maybanks
