Upload
others
View
6
Download
0
Embed Size (px)
Citation preview
UX ResearchQuestionnaires / Surveys
KP Ludwig John
UX Researchmethods Questionnaire
UX ResearchQuestionnaires / Surveys
KP Ludwig John
Research Methods
Qualitative Quantitative•describes the qualities or characteristics of something
•provide details about human behavior, emotion, and personality characteristics
•helps to understand why people do the things they do
•any research that can be measured
•provide data that can be expressed in numbers
•answers questions like “How many people clicked here”
Reference: http://www.uxmatters.com/mt/archives/2012/09/strengths-and-weaknesses-of-quantitative-and-qualitative-research.php
Questionnaire?
Approaches
UX ResearchQuestionnaires / Surveys
KP Ludwig John
Reference: http://www.fao.org/docrep/w3241e/w3241e05.htm
QuestionnaireApproaches
Example: Interviewing the head of the household to find out how decisions are made within the family when purchasing breakfast foodstuffs
A formal questionnaire may restrict the discussion and prevent a full exploration of the processes.
Prepare a brief guide, listing perhaps ten major open-ended questions, with appropriate probes/prompts listed under each.
Exploratory questionnaire: • collection of qualitative data • no statistically evaluation • probably no formal questionnaire needed
UX ResearchQuestionnaires / Surveys
KP Ludwig John
Reference: https://chaione.com/blog/ux-research-standardizing-usability-questionnaires/
QuestionnaireApproaches
Benefits: Quantification: Standardized measurements
Generalization: generalizing a finding from a sample to the greater population.
Communication: easier to communicate findings by referring to standardized metrics.
Quick Comparisons: easy to compare different iterations throughout the development process.
Formal standardized questionnaire: • to test and quantify hypothesis • to be analyzed statistically
UX ResearchQuestionnaires / Surveys
KP Ludwig John
StandardizedQuestionnaires
UX ResearchQuestionnaires / Surveys
KP Ludwig John
System Usability Scale (SUS)
https://en.wikipedia.org/wiki/System_usability_scale
Developed by John Brook in 1986 at Digital Equipment Corporation (UK) as a tool in usability engineering of electronic office systems
simple, ten-item attitude testgiving a global view of subjective assessments of usability
Likert scale
Summed up in form of a
UX ResearchQuestionnaires / Surveys
KP Ludwig John
System Usability Scale (SUS)
https://en.wikipedia.org/wiki/Likert_scale
Respondents specify their level of agreement or disagreement on a symmetric agree-disagree scale for a series of statements.
bipolar scaling method, measuring either positive or negative response to a statement
Likert scalenamed after its inventor, psychologist Rensis Likert (1903 - 1981)
Five response options for respondents; from Strongly agree to Strongly disagree.
SUS: Question 1
I think I would like to use this system frequently
http://www.usabilitest.com/sus
UX ResearchQuestionnaires / Surveys
KP Ludwig John
System Usability Scale (SUS)The 10 template questions which you can adapt:
1. I think that I would like to use this system frequently. 2. I found the system unnecessarily complex. 3. I thought the system was easy to use. 4. I think that I would need the support of a technical person
to be able to use this system. 5. I found the various functions in this system were well integrated. 6. I thought there was too much inconsistency in this system. 7. I would imagine that most people would learn to use
this system very quickly. 8. I found the system very cumbersome to use. 9. I felt very confident using the system. 10. I needed to learn a lot of things
before I could get going with this system.
https://usabilitygeek.com/how-to-use-the-system-usability-scale-sus-to-evaluate-the-usability-of-your-website/
UX ResearchQuestionnaires / Surveys
KP Ludwig John
System Usability Scale (SUS)Calculate the ScoreUsers have ranked each of the 10 templates questions from 1 to 5
• For each of the odd numbered questions, subtract 1 from the score.
• For each of the even numbered questions, subtract their value from 5.
• Take these new values which you have found, and add up the total score. Then multiply this by 2.5.
The result is your score out of 100. (This is NOT a percentage!)
https://usabilitygeek.com/how-to-use-the-system-usability-scale-sus-to-evaluate-the-usability-of-your-website/
What does the score tell?
UX ResearchQuestionnaires / Surveys
KP Ludwig John
System Usability Scale (SUS)
https://usabilitygeek.com/how-to-use-the-system-usability-scale-sus-to-evaluate-the-usability-of-your-website/
Interpret the ScoreTranslated into notes:
https://measuringu.com/sus/
• 80.3 or higher is an A. People love your site andwill recommend it to their friends
More details at:
• 51 or under gets you a big fat F. Make usability your priority now and fix this fast.
• 68 or thereabouts gets you a C. You’re doing OK but could improve
SUS scores from about 500 surveys, conducted by Jeff SauroAverage SUS score = 68
UX ResearchQuestionnaires / Surveys
KP Ludwig John
User Experience Questionnaire (UEQ)
http://www.cheval-lab.ch/was-ist-usability/usabilitymethoden/frageboegen/user-experience-questionnaire/
Originally developed at SAPfreely accessible since 2006
Explores the following six factors:
Used to measure the user experience of software products Takes into account classic aspects of Usability (Efficiency) as well as of User Experience (Attractiveness)
UX ResearchQuestionnaires / Surveys
KP Ludwig John
User Experience Questionnaire (UEQ)
http://www.ueq-online.org/
(Attraktivität)Overall Impression of the product.
Do users like or dislike this?
Explores the following six factors:
(Verständlichkeit)Is it easy to get familiar
with the product?
(Effektivität)Can users solve their tasks
with the product without unnecessary effort?
(Verlässlichkeit)Does the user feel in control
of the interaction?
(Stimulation)Is it exciting and motivating
to use the product?
(Originalität)Is the product
innovative and creative?
UX ResearchQuestionnaires / Surveys
KP Ludwig John
User Experience Questionnaire (UEQ) http://www.ueq-online.org/
Explores the following six factors:
Attractivenessis a pure valence (Wertigkeit) dimension
Perspicuity, Efficiency, Dependability pragmatic quality aspects
(goal-directed)
Stimulation, Novelty hedonic (Sinnengenuss) quality aspects
(not goal-directed)
Grouped in categories
UX ResearchQuestionnaires / Surveys
KP Ludwig John
User Experience Questionnaire (UEQ)
26 pairs of contrasting attributes that may apply to the product
http://www.ueq-online.org/
Questionnaire
UX ResearchQuestionnaires / Surveys
KP Ludwig John
User Experience Questionnaire (UEQ)
Please decide spontaneously. Don’t think too long about your decision to make sure that you convey your original impression
Tick a circle in every line. It is your personal opinion that counts.
http://www.ueq-online.org/
Procedure
UX ResearchQuestionnaires / Surveys
KP Ludwig John
User Experience Questionnaire (UEQ)Typical applications
http://www.ueq-online.org/
Compare the UX of two products
However, if you want to draw conclusions on this (especially if your sample is small) you have to check if the differences are significant.
Version A shows for all scales, with the exception of Novelty better values than version B.
UX ResearchQuestionnaires / Surveys
KP Ludwig John
User Experience Questionnaire (UEQ)Typical applications
General UX Evaluation of a product
http://www.ueq-online.org/
Haptic Interface
Research Topics:Do Users understand how to interact / what to do?
Interactive Mirror
Research Topics:Dimension, Observer DistanceObserver Position
Source: IA6 2015, Team Salz, IA Installation
UX ResearchQuestionnaires / Surveys
KP Ludwig John
User Experience Questionnaire (UEQ)Typical applications
General UX Evaluation of a product
http://www.ueq-online.org/
Does the product fulfill the general expectations concerning UX? Such expectations of users are formed by products they frequently use.
Source: IA6 2015, Team Salz, IA Installation
UX ResearchQuestionnaires / Surveys
KP Ludwig John
User Experience Questionnaire (UEQ)Typical applications
http://www.ueq-online.org/
General UX Evaluation of a productDoes the product fulfill the general expectations concerning UX? Such expectations of users are formed by products they frequently use.
Source: IA6 2015, Team Salz, IA Installation
UX ResearchQuestionnaires / Surveys
KP Ludwig John
What should be changed?Cannot be answered directly by a quantitative measurement of user experience.A connection of product features to the measurement is required.
UEQ shows a pattern of 6 measured user experience qualities. Serves as Basis to make some assumptions where to look for improvements.
User Experience Questionnaire (UEQ)Benchmark graph
http://www.ueq-online.org/
Shows UX quality of evaluated product
UX ResearchQuestionnaires / Surveys
KP Ludwig John
CustumizedQuestionnaire
(Survey)
UX ResearchQuestionnaires / Surveys
KP Ludwig John
Source: http://www.surveysystem.com/sdesign.htm
Survey
Basic types:
Questions and Answers
UX ResearchQuestionnaires / Surveys
KP Ludwig John
Source: https://www.nngroup.com/articles/open-ended-questions/
Survey in User Research
• allow to give a free-form answer • users can to talk in depth about a question • possible to derive valid information from small number of users
• can be answered with “Yes” or “No“ • have a limited set of possible answers • good for surveys
Closed-ended questions
Open-ended questions
• get higher response rates (easier to answer) • useful with larger number of participants • responses can easily be analyzed statistically
UX ResearchQuestionnaires / Surveys
KP Ludwig John
Source: https://www.nngroup.com/articles/open-ended-questions/
Survey
Closed / Open - How to ask?
closed open
Are you satisfied? How satisfied or dissatisfied are you with this process?
Did it act as you expected? What would (did) you expect to happen when you ... ?
Do you think you would use this? How would this fit into your work?
Does that work for you? What do you think about that?
Do you normally … ? How do you normally ... ?
Do you like this?What would you most want to change about … ? Which things did you like the best about … ?
in User Research
UX ResearchQuestionnaires / Surveys
KP Ludwig John
Source: https://www.nngroup.com/articles/open-ended-questions/
Survey
Closed / Open - How to ask?
closed open
Are you satisfied? How satisfied or dissatisfied are you with this process?
Did it act as you expected? What would (did) you expect to happen when you ... ?
Do you think you would use this? How would this fit into your work?
Does that work for you? What do you think about that?
Do you normally … ? How do you normally ... ?
Do you like this?What would you most want to change about … ? Which things did you like the best about … ?
in User Research
2 minutes
UX ResearchQuestionnaires / Surveys
KP Ludwig John
Source: https://www.nngroup.com/articles/open-ended-questions/
Survey
Closed / Open - How to ask?
closed open
Are you satisfied? How satisfied or dissatisfied are you with this process?
Did it act as you expected? What would (did) you expect to happen when you ... ?
Do you think you would use this? How would this fit into your work?
Does that work for you? What do you think about that?
Do you normally … ? How do you normally ... ?
Do you like this?What would you most want to change about … ? Which things did you like the best about … ?
in User Research
UX ResearchQuestionnaires / Surveys
KP Ludwig John
Source: https://www.nngroup.com/articles/open-ended-questions/
Survey
people may
• share motivations that you didn’t expect • mention behaviors and concerns that you knew nothing about. • reveal surprising mental models, problem-solving strategies, hopes, fears, and much more.
Open-ended questionsallow you to find more than you anticipate
in User Research
UX ResearchQuestionnaires / Surveys
KP Ludwig John
Source: https://www.nngroup.com/articles/open-ended-questions/
Survey
Closed-ended questions stop the conversation and eliminate surprises
Choose your favorite ice cream: vanilla, strawberry, or chocolate.
Answers that you suggest can reveal what you are looking for. People may be influenced by the questions.
When you ask closed-ended questions, ask an open-ended question at the end:“What else would you like to say about that?”
in User Research
Ad Other __________
to a set of multiple-choice answers … to get open-ended feedback.
UX ResearchQuestionnaires / Surveys
KP Ludwig John
Survey
When to Ask Open-Ended Questions
in User Research
While conducting conceptual research:
• Which problems to solve • What kind of solution to provide • Who to design for
For exploratory studies, such as
• Qualitative usability testing • Interviews and Diary studies • Persona research • Task analysis
Source: https://www.nngroup.com/articles/open-ended-questions/
UX ResearchQuestionnaires / Surveys
KP Ludwig John
Survey
When to Ask Closed-Ended Questions
in User Research
• In quantitative usability studies, measuring time on task and error rates, to compare results among users
• In surveys which large number of participants
• When the set of answers is strictly limited for some reason
Source: https://www.nngroup.com/articles/open-ended-questions/
UX ResearchQuestionnaires / Surveys
KP Ludwig John
Source: http://www.surveysystem.com/sdesign.htm
treated as
Rating Scales / Agreement Scales
multiple choice
multiple choice
numeric
Survey Questions and Answers
UX ResearchQuestionnaires / Surveys
KP Ludwig John
Source: http://www.surveysystem.com/sdesign.htm
Structure / OrderEarly questions in a survey should be easy and pleasant to answer. These kinds of questions encourage people to continue the survey.Leave difficult or sensitive questions until near the end of your survey
Survey Questions and Answers
Whenever there is a logical or natural order to answer choices, use it. Always present agree-disagree choices in that order. Presenting them in disagree-agree order will seem odd. Same for positive to negative and excellent to poor
When using numeric rating scales higher numbers should mean a more positive or more agreeing answer
UX ResearchQuestionnaires / Surveys
KP Ludwig Johnhttps://youtu.be/bi2JMRpL9Ik
Survey
UX ResearchQuestionnaires / Surveys
KP Ludwig John
Source: https://youtu.be/bi2JMRpL9Ik by Chris Gray
How to Create?
Step 1What are your objectives for the survey?Formulated research topics / field
Step 2Find questions to cover them
Step 3Group similar questionsOrder them logically
Step 4Pre-Test your survey with a small sub-group of your audience
Survey
UX ResearchQuestionnaires / Surveys
KP Ludwig John
Questions - How to phrase?• easy to understand
• appropriate for your audience
Source: http://uxmastery.com/better-user-research-through-surveys/ by Chris Gray
!This doesn’t mean that they are qualified to answer them, or are able to provide insightful feedback.
People can and will answer just about any question put in front of them
Survey
UX ResearchQuestionnaires / Surveys
KP Ludwig John
Questions - How to phrase?
• One Issue (concept) per questionDon’t combine two in one
(Does climate change exist + politics do not care)
Source: http://uxmastery.com/better-user-research-through-surveys/ by Chris Gray
• avoid double negatives or confusing questions
(Don’t you think, this isn’t true?)
Survey
UX ResearchQuestionnaires / Surveys
KP Ludwig John
• use balanced rating scales (positive - negative)
Source: http://uxmastery.com/better-user-research-through-surveys/ by Chris Gray
Survey Questions - How to phrase?
UX ResearchQuestionnaires / Surveys
KP Ludwig John
• use open questions
Not: Do you think, that … yes/noBut: To what extend do you agree with the following …
• allow „Don’t Know“ as multiple choice option
• combine multiple choice with Text field for explanations (why, comments)
Source: http://uxmastery.com/better-user-research-through-surveys/ by Chris Gray
Survey Questions - How to phrase?
keep the survey short!
UX ResearchQuestionnaires / Surveys
KP Ludwig John
CustumizedQuestionnaire
(Interview)
UX ResearchQuestionnaires / Surveys
KP Ludwig John
Source: https://www.nngroup.com/articles/leading-questions/
Interviews Questions - How to phrase?
Avoid Leading Questions
Which phrasing of a follow-up question will result in more accurate results? ?
Discuss and collect1 minute
• “What was easy or difficult about getting to the content you wanted?”
• “Why did you have difficulty with the navigation?”
• “I saw you were having difficulty with the navigation. What happened?”
UX ResearchQuestionnaires / Surveys
KP Ludwig John
Source: https://www.nngroup.com/articles/leading-questions/
Questions - How to phrase?
Avoid Leading Questions
• “What was easy or difficult about getting to the content you wanted?”
• “Why did you have difficulty with the navigation?”
• “I saw you were having difficulty with the navigation. What happened?”
The first two questions are leading,meaning that it includes or implies the desired answer to the question in the phrasing of the question itself.
Which phrasing of a follow-up question will result in more accurate results?
Interviews
UX ResearchQuestionnaires / Surveys
KP Ludwig John
Source: https://www.nngroup.com/articles/leading-questions/
Questions - How to phrase?
Avoid Leading Questions
• “I saw you were having difficulty with the navigation. What happened?”
The interviewer rephrases what was observed, which may not be an accurate representation of the user’s experience. The question also names a user interface element – the navigation – which is a term that users may or may not fully understand, relate to, or normally use.
Question 1
Interviews
UX ResearchQuestionnaires / Surveys
KP Ludwig John
Source: https://www.nngroup.com/articles/leading-questions/
Questions - How to phrase?
Avoid Leading Questions
Again, this question implies the answer and assumes that navigation was the problem.
It also puts the blame on the user, rather than on the site. The question focusses on the user’s actions as opposed to the elements in the site that may have contributed to the user’s actions.
Question 2
• “Why did you have difficulty with the navigation?”
!
Interviews
UX ResearchQuestionnaires / Surveys
KP Ludwig John
Source: https://www.nngroup.com/articles/leading-questions/
Questions - How to phrase?
Avoid Leading Questions
Steers the user to the topic of interest without suggesting terms or feelings to the user.
The user can say it was simple to move around or difficult, without disagreeing with the interviewer. Here the interviewer offers a general frame for the topic of the question, rather than suggesting a response.
Question 3
• “What was easy or difficult about getting to the content you wanted?”
!
Interviews
UX ResearchQuestionnaires / Surveys
KP Ludwig John
Source: https://www.nngroup.com/articles/leading-questions/
Questions - How to phrase?
Avoid Leading Questions
Leading questions
The interviewer is the “authority” in the room and many participants will not want to disagree
… make it difficult for the participant to express another opinion.
… interject the answer we want to hear in the question itself.
… may result in biased or false answers
… rob us of the opportunity to hear an insight we weren’t expecting
Interviews
UX ResearchQuestionnaires / Surveys
KP Ludwig John
Source: https://www.nngroup.com/articles/leading-questions/
Questions - How to phrase?
Avoid Leading QuestionsTherefore
Interviews
• Do not rephrase in our own words. ◦ Participant: “I notice this picture here… “ ◦ Researcher: “You mentioned that the picture was helpful. What about it did you like? “ ◦ Improvement: “You mentioned the picture…?”
• Do not assume you know what the user is feeling. ◦ “When you were struggling with this task, what was happening?” ◦ Improvement: “What was easy or difficult about completing that task?”
• Do not name an interface element. ◦ “The related links on the side of the page here – where would those lead?” ◦ Improvement: “This area on the side of the page… [point to area]. What is that?”
• Do not suggest an answer. ◦ “How well would this save time for you during your workday?” ◦ Improvement: “How might this affect your efficiency, if at all?”
UX ResearchQuestionnaires / Surveys
KP Ludwig John
Source: https://www.interaction-design.org/literature/article/how-to-prepare-for-a-user-interview-and-ask-the-right-questions
Questions - OrderInterviews
Ann Blandford - professor of Human Computer Interaction at University College London
Which questions at the beginning of an Interview
UX ResearchQuestionnaires / Surveys
KP Ludwig John
Source: https://www.interaction-design.org/literature/article/how-to-prepare-for-a-user-interview-and-ask-the-right-questions
Questions - OrderWhich questions at the beginning of an Interview
Interviews
Start with brief and easy to understand questions in a vocabulary that is familiar to your participant
Turn your overall research question into more concrete questions that will then answer your overall question
UX ResearchQuestionnaires / Surveys
KP Ludwig John
Source: https://www.interaction-design.org/literature/article/how-to-prepare-for-a-user-interview-and-ask-the-right-questions
Questions - DetailInterviews
Ann Blandford - professor of Human Computer Interaction at University College London
Concrete Examples of critical incidents
UX ResearchQuestionnaires / Surveys
KP Ludwig John
Source: https://www.interaction-design.org/literature/article/how-to-prepare-for-a-user-interview-and-ask-the-right-questions
Interviews
Use questions like “Can you tell me about the last time you used video streaming?”
Ask the concrete “how” and “what” questions before you ask the more abstract “why” questions
Ask for concrete examples.
Concrete Examples of critical incidents
Questions - Detail
UX ResearchQuestionnaires / Surveys
KP Ludwig John
Source: http://uxmastery.com/better-user-research-through-surveys/ by Chris Gray
When to use Surveys?
• Who your users are• What your users want• What they purchase• Where they shop• What they own• What they think of your brand or product
Effective method for identifying:
Generally: to understand the user
Survey
UX ResearchQuestionnaires / Surveys
KP Ludwig John
Source: http://uxmastery.com/better-user-research-through-surveys/ by Chris Gray
When to use Surveys?
• Gathering feedback on a live product or during a pilot
• Exploring the reasons people visit a website and assessing their experience of that visit
• Quantifying results from qualitative research activities
• Evaluating usability
In UX context
Survey
UX ResearchQuestionnaires / Surveys
KP Ludwig John
Source: http://uxmastery.com/better-user-research-through-surveys/ by Chris Gray
When to use Surveys?
But be
Cautious asking about behavior!
For assessing the effectiveness of a public transport ticketing system,observing people buying tickets and traveling throughout the network would yield more accurate and useful feedback than asking people how they had used the system over the last week.
Because
Survey
UX ResearchQuestionnaires / Surveys
KP Ludwig John
https://www.nngroup.com/articles/interviewing-users/
Because
The critical failing: You're asking people to either remember past use or speculate on future use of a system.
Human memory is fallible. People can't remember the details of how they used a website.They tend to make up stories …
What users say and what they do are different- Jacob Nielsen
UX ResearchQuestionnaires / Surveys
KP Ludwig John
https://www.nngroup.com/articles/interviewing-users/
What users say and what they do are different- Jacob Nielsen
Because
This is why Agile development and paper prototyping methods are valuable.
When users have something concrete to interact with, it's usually obvious when you're solving their problems in a way that's easy and pleasant to work with — and equally obvious when you're not.
Users are pragmatic and concrete.They typically have no idea how they might use a new technology based on a description alone.
UX ResearchQuestionnaires / Surveys
KP Ludwig John
Survey ToolsMany online tools available Explore and try them out!
survio.com/de
UX ResearchQuestionnaires / Surveys
KP Ludwig John
Survey Tools
survio.com/de
UX ResearchQuestionnaires / Surveys
KP Ludwig John
Practice Finalize and plan
• Research Question and measurementsWhat is your research topic?What do you want to find out?Which measurements will be used?
• Two versions of the survey• one biased
indicate the biases usedand the dedicated results / tendency
• one contra biased
UX ResearchQuestionnaires / Surveys
KP Ludwig John
Practice Criteria
• Precise definition ofResearch Question and measurements
• definition of biases and striven tendency
• Methodically consistent implementation and execution
• Critical Evaluation of test procedure
• Clear summery of findings
UX ResearchQuestionnaires / Surveys
KP Ludwig John
PracticeNext week
• Topic of your research • dedicated results of surveys • How to achieve these results (Biases) • How to conduct your survey practically
hs-augsburg.de/homes/john
Next appointments and script:
Discussion and Feedback