Webinar:
Remote Testing Methods & Tools
December 13th, 2011
Bill Albert, Director of Design & Usability Center at Bentley University presents a complimentary webinar on Remote Testing Methods and Tools.
With special guests:
2
Webinar Overview
1. Brief introductions & goal of the webinar
2. Updated overview of competitive landscape, updated figures, etc
3. The different methods / categories / tools. When to use what?
4. Types of research goals
1. Categories of tools available
2. Overview of some of those tools
5. Examples:
– Usertesting.com
– Optimal Workshop
– UserZoom
6. Takeaways, Q&A
3
• Speakers today:o Bill Albert, Director of Design & Usability Center, Bentley Universityo Dave Garr, Co-Founder, UserTesting.como Andrew Mayfield, CEO, Optimal Workshopo Matt Paulus, Director of US Sales, UserZoom
• Goals of today’s session
Introduction
[email protected] 781.891.2500 | www.bentley.edu/usability
Bill AlbertDecember 13, 2011
Remote Testing Methods and Tools
5
• Introduction to remote usability testing
• Quantitative-based tools
• Qualitative-based tools
Agenda
6
• Director, Design and Usability Center, Bentley University• Experience with remote testing tools• I am data driven!
About me
7
I am a big fan of usability testing in the lab!
Introduction to remote usability testing
9
What do we mean by “remote testing tools”?• A remote testing tool is any technology that allows a researcher to collect
data from users about their experience in using an interface, without direct, face-to-face contact.
There are two ways of organizing remote testing tools• Moderated vs. Unmoderated• Qualitative vs. Quantitative-based
Defining Remote Testing Tools
Moderated Unmoderated
Qualitative-based Qualitative-based
Quantitative-based Quantitative-based
10
Unmoderated Testing Tools
Unmoderated tools
Qualitative-based
Video tools
Reporting
Expert Reviews
Quantitative-based
Full-service
Self-service
Card Sorting/IA
Surveys
Click/Mouse
11
Our UX Toolkit
12
• Saves timeo Lab study takes 2-4 weeks from start to finish, unmoderated typically takes
hours to a few days*
• Saves moneyo Participants compensation typically a lot less ($10 vs. $100)o Tools are becoming very inexpensive
• Reliable metricso Only (reasonable) way to collect UX data from large sample sizes
• Geography is not a limitationo Collect feedback from customers all over the world
• Greater Customer insighto Richest dataset about the customer experience
Why Should You Care?
13
• Questions from senior management are becoming more complex and time sensitive
• Traditional usability is no longer enough
• Push to measure the UX
• Convergence with market research and web analytics to paint a more complete picture of the UX
• Budgets/timeline constraints
Why is this important NOW?
14
Which one goes first?
Lab first, then Unmoderated Unmoderated first, then Lab
Identify/fix “low hanging fruit”, then focus on remaining tasks with large sample size
Identify the most significant issues online through metrics, then use lab study to gather deeper qualitative understanding of those issues
Generate new concepts, ideas, questions through lab testing, then test/validate online
Collect video clips or more quotes of users to help bring metrics to life
Validate attitudes/preferences observed in lab testing
Gather all the metrics to validate design – if it tests well, then no need to bring users into the lab
Quantitative-based tools
16
Unmoderated Testing Tools - Quantitative
Unmoderated tools
Qualitative-based
Video tools
Reporting
Expert Reviews
Quantitative-based
Full-service
Self-service
Card Sorting/IA
Surveys
Click/Mouse
17
Common Research Questions:• What are the usability issues, and how big?• Which design is better, and by how much?• How do customer segments differ?• What are user design preferences? • Is the new design better than the old design?• Where are users most likely to abandon a transaction?
Types of Studies:• Comprehensive evaluation• UX benchmark• Competitive evaluation• Live site vs. prototype comparison• Feature/function test• Discovery
Overview
Typical Metrics:• Task success• Task time• Self-report ratings such as ease of
use, confidence, satisfaction• SUS• Click paths• Abandonment
18
Strengths:• Comparing products• Measuring user experience• Finding the right participants• Focusing on design improvements• Insight into the user’s real
experience
Limitations:• Not well suited to rapid, iterative
design• Need a deep understanding of
issues• Studies that require long sessions• Lose control over prototypes• Internet access
Strengths / Limitations / Myths
Myths:• Only test with websites• Very expensive• Only gather quantitative data• A lot of noise in the data• Same as any market research study• Comparing products
19
Full Service Tools – Online Usability Testing
20
Self Service Tools – Online Usability Testing
21
Card Sorting / IA Tools
22
Online Surveys
23
Click and Mouse Tools
Qualitative-based tools
25
Unmoderated Testing Tools - Qualitative
Unmoderated tools
Qualitative-based
Video tools
Reporting
Expert Reviews
Quantitative-based
Full-service
Self-service
Card Sorting/IA
Surveys
Click/Mouse
26
OverviewCommon Research Questions:• What are the big pain usability pain points, and
wins?• How do users think, and feel about the design?• Are we on the right (design) track? • What is the overall experience like? • Why might users abandon a transaction?
Types of Studies:• Low fidelity prototype• High fidelity prototype• Competitive evaluation• Comprehensive evaluation• Feature/function test• Discovery
Common Metrics:• Frequency of issues• Verbatim comments• Task success (via video)• Task time (via video)
27
Video Tools
28
Report Tools
29
Expert Review
30
Online:www.measuringUX.comwww.measuringusability.comhttp://usability.bentley.eduhttp://www.usefulusability.com/24-usability-testing-tools/http://remoteresear.ch/tools/
Follow me:@UXMetrics@BentleyDUC
LinkedIn Group:Bentley Design and Usability Center
Additional Resources
Dave GarrCo-Founder
UserTesting.com
In Cyberspace, no one can hear your visitor scream.
What’s your visitor thinking?
33
Solution?
How It Works
“I don’t know why it keeps doing this.This is definitely very confusing and frustrating.”
Write an Annotation
“I can’t figure out which cartridge goes with my printer”
How to share clips
View Tester’s Answers
Follow up with users
Export to Excel
42
Mobile Testing
• Test mobile apps
• Test mobile-friendly sites
43
What do you want to test?
“I’d like to look at these three star reviews, … it doesn’t let me do that… I’d have to go through all of the
reviews.”
44
Presenting Your Findings
Andrew MayfieldCEO
Optimal Workshop
Quantitative analysis to assess the
effectiveness of an Information
Architecture is
hard work.
Peter Morville’sUX Honeycomb
Optimal Workshop
The Suite
by Optimal Workshop
Qualitativevs
Quantitative
www.optimalworkshop.com/blogSee latest blog post on this topic
Par
ticip
ant
Cen
tric
Ana
lysi
s
Safe orexistingoptions
Radicaloption
?
BENCHMARK
Remote studies enable near unlimitediteration and participation
Matt PaulusDirector of US Sales
UserZoom
UserZoom’s Software Solutions
• On-demand, web-based SaaS
• 100% self-service
• All-in-one quantitative unmoderated online research
• Enterprise level software
• Feature rich & intuitive
• Quantitative remote testing
• Natural context
• International/domestic testing
• No moderation needed
• Behavior (analytics) + Feedback (online Surveys)
Unmoderated Remote Testing Methodology
UserZoom Products
UserZoom Products
60
Live Websites Wireframes & Prototypes
Multimedia content
Mobile Interfaces & Apps
Compatible Platforms
UserZoom Task-Based Usability Testing
UserZoom Results – Open Ended/Text Clouds
UserZoom Results – Net Promotor Score
UserZoom Results – Filters/Segmentation
UserZoom Results – Effectiveness & Efficiency
UserZoom Results – Clickstreams
UserZoom Results- Click Heatmaps
68
• Remote testing tools are becoming more advanced every day, and are a critical part of the UX toolkit
• Find the right type of tool based on your research question
• Demo a select number of remote testing tools to find the one that will deliver the results you need, within budget and timeframe
Key Takeaways
69
Quick Poll!
Thank You!
Floor open to Q&A
Bill AlbertDirector of Design & Usability CenterBentley University
Dave GarrCo-Founder
UserTesting.com
Andrew MayfieldCEO
Optimal Workshop
Matt PaulusDirector of US
SalesUserZoom