View
2
Download
0
Category
Preview:
Citation preview
1
CS5760
Graduate Human-Computer Interactions & Usability
Evaluation Assignment 6
Usability Test Report
Jiban Adhikary (Grad 9)
Team 2 (Blu Team)
April 30, 2018
2
Table of Contents:
1 Introduction 3
2 Undergraduate System 3
3 Usability Test Plan 3
4 Results 8
5 Conclusion 17
Appendix A (Undergrad student attendance) 19
Appendix B (Bug report) 20
Appendix C (Consent Form) 21
Appendix D (Pre-test Questionnaire) 22
Appendix E (Post-test Questionnaire, Scenario 1) 23
Appendix F (Post-test Questionnaire, Scenario 2) 24
Appendix G (Post-test Questionnaire, Scenario 3) 25
Appendix H (Post-test Questionnaire, General) 26
3
1 Introduction
This documentation provides a detailed usability test report for a smartphone app. First, it provides a summary of the app or software that is being developed by a team of undergraduate students, the Blu Team. Then it outlines the usability test plan. Next, it discusses the usability test results. The documentation also includes some forms in the Appendix sections. These forms were used throughout the test process. If you have read the Evaluation Assignment 4, then you can skip reading the sections 2 to 3.
2 Undergraduate System
Blu Team is working on an app which is basically a Volunteer Monitoring app. Volunteers are the backbone of Oklahoma Blue Thumb program, the eyes and ears of creeks and streams across the state. By using the app the volunteers will monitor the creeks monthly. They will assess the physical habitat, test sample of the creek’s water. The monitoring will take place at the creek, and all the data related to environmental or physical factors affecting the creeks will be collected. It will also involve collecting a sample of water which will later be chemically tested in the lab. The chemical tests are supposed to be held on a comfortable time the volunteer prefers to do. Previously all of this data were sent out to Blue Thumb using traditional media like fax, email, or snail mail. This app will ease the monitoring process by facilitating volunteers to submit data using their smartphones and email them to the quality assurance officer. The app is now ready to run some user trial and this documentation provides the procedural way to conduct those user trial.
3 Usability Test Plan
The usability test plan events included the following:
I.Welcome / Introductory Events II.Signing of the consent form
III.Pre Test questionnaire / Demographics IV.Carrying out the test tasks V.Post Test Questionnaire
3.1 Welcome / Introductory Events
The experimenter welcomed the participant and had the participant sit at a desk or table. The participant sat at this location while they completed initial paperwork.
The experimenter described the objective of the app and why it had been developed. The experimenter ascertained the participant that there were no risks that could possibly be risen from running the usability tests. The experimenter explicitly asked the participant if there was any question and answered if there was any.
4
The experimenter also mentioned that the data being collected will be kept confidential and any link to trace the identity of the participant will be destroyed.
3.2 Signing of The Consent Form
The experimenter handed out the consent form to the participant. The participant read the consent form and after signing it he returned it to the experimenter. The experimenter answered if there was any question asked by the participant. A sample consent form is included in Appendix C.
3.3 Pre Test Questionnaire
Before the usability task tests, the participant filled out a demographic/ pre - test questionnaire. A sample pre-test questionnaire has been included in Appendix D.
3.4 Carrying Out The Test Tasks
Three of the following three scenarios were evaluated during the test. After each of the test tasks had been completed, the participant filled out a questionnaire specific to the scenario completed.
3.4.1 Test Scenario 1: Visibility of icons and texts
3.4.1.1 Goals:
To check if the icons used in the app are of proper sizes To check if the font size of the texts in the app are appropriate To check whether the background color and identical buttons have consistent
color
3.4.1.2 Quantitative measurement:
Number of cases where the font size seemed too big or too small Number of cases where the color of buttons or texts or background seemed
inconsistent Approximate amount of time to interact and get response from the app Number of bugs encountered around the process
3.4.1.3 Scenario description: The participant was asked to launch the app from the beginning. He was asked to navigate through the pages and fill in some random fields. He was advised to keep some mental notes about the icons and text labels used in the app while he was navigating through. He was asked to locate any button or text at random saying like - “Can you locate the next button?”
3.4.1.4 Task list: The participant did the following tasks -
5
Launch the app Navigate through the pages Keep mental notes of the texts and icons Report
3.4.1.5 Qualitative measurement list:
How difficult was it to find a specific button or text? Did the user find any difficulties to read any text?
3.4.1.6 Potential observations of participant:
Facial expression of the participant Was participant moving his head back and forth occasionally? Was the distance between the smartphone and participant’s eyes inconsistent? If the participant asking questions?
3.4.1.7 Bug Report Form: A sample bug report form is presented in Appendix B. Multiple forms was used if necessary.
3.4.1.8 Post Scenario interview or questionnaire questions:
Scenario specific questions: Questionnaire for scenario 1 can be found in Appendix E
3.4.1.8 Test set up details: No specific set up was necessary. A participant was asked to use his smartphone and was given proper suggestions to find out the app icon.
3.4.2 Test Scenario 2: Alignment, text wrapping and natural flow of app pages
3.4.2.1 Goals:
To check if form labels and input fields have been properly aligned To check whether input text is wrapped within the input field To check if there is a consistent flow of pages while navigating back and forth
3.4.2.2 Quantitative measurement:
Number of cases where the texts or icons were not properly aligned Number of cases where user could not navigate to next or previous page Number of cases where texts were not wrapped for a large block of input text Number of events where user could not slide up and down if necessary Number of bugs encountered around the process
3.4.2.3 Scenario description: The participant was asked to launch the app. He was asked to notice the text and object alignment and was asked to enter a block of paragraph in an input field like comment section. He was also asked to navigate back and forth or up and down
6
3.4.2.4 Task list: The participant did the following -
Launch the app Notice the alignments Enter a block of texts Navigate back and forth or up and down
3.4.2.5 Qualitative measurement list:
Which form of alignment was appropriate Was the sliding up and down bar visible
3.4.2.6 Potential observations of participant:
Facial expression of the participant Finger movements of the participant while sliding up and down or navigating
back and forth If the participant asking questions?
3.4.2.7 Bug Report Form: A sample bug report form is presented in Appendix D. Multiple forms will be used if necessary.
3.4.2.8 Post Scenario interview or questionnaire questions:
Scenario specific questions: Questionnaire for scenario 2 can be found in Appendix F
3.4.2.9 Test set up details: No specific set up was necessary. A participant was asked to use his smartphone and was given proper suggestions to find out the app icon.
3.4.3 Test Scenario 3: Text input validation, feedback and exceptions
3.4.3.1 Goals:
To check if the app validates user input for range specific values To check what kind of feedback the app provides To check if the app has a fail-safe mechanism in case of an unusual event
3.4.3.2 Quantitative measurement:
Number of text fields where user can input arbitrary data Number of events a user is not provided with any feedback while trying to input
invalid data Number of bugs encountered around the process
3.4.3.3 Scenario description: The participant was asked to start up the app. He was asked to interact with the help functions. He was also asked to enter some arbitrary values in the input fields. Finally, he was asked to tilt or shake the smartphone or enable/disable the wifi.
3.4.3.4 Task list: The participant did the following -
7
Launch the app Try the help functions or labels with ‘?’ mark on it Enter arbitrary input Shake/tilt the phone, or enable/disable wifi etc
3.4.3.5 Qualitative measurement list:
Did the app crash if it is tilted/shaken or wifi is enabled/disabled
3.4.3.6 Potential observations of participant:
Facial expression of the participant If the participant asking questions?
3.4.3.7 Bug Report Form: A sample bug report form is presented in Appendix B. Multiple forms will be used if necessary.
3.4.3.8 Post Scenario interview or questionnaire questions:
Scenario specific questions: Questionnaire for scenario 3 can be found in Appendix G
3.4.3.9 Test set up details: No specific set up was necessary. A participant was asked to use his smartphone and was given proper suggestions to find out the app icon.
3.5 Post Test Questionnaire
The participant filled out a general questionnaire after the above scenarios had been completed. The general questionnaire is included in Appendix H.
8
4 Results
4.1 Pre Test Questionnaire Results
Pre-test questionnaire results are depicted in the following figures.
Uses smartphone for
Talking to people Texting Emailing Playing games Web browsing
Figure 4.1: Pre-test questionnaire results
4.2 Test Results on Scenario 1: Visibility of icons and texts
The post-test questionnaire conducted after the first test scenario reflects that the users liked the UI elements. According to their thought UI elements were precise and appropriate.
40%
60%
0%
Smartphone OS
iOS
Android
Others 1 1
3
smaller than4.7
4.7-5.2 larger than5.2
Screen sizes of the used smartphones (inches)
0
1
2
3
4
5
less than ayear
1-2 years 2-3 years more than3 years
Smartphone usage
Prior experience to survey water body
Yes No
Prior experience to run a chemical test
Yes No
9
Figure 4.2.1: User satisfaction on the size of the texts
Figure 4.2.2: User satisfaction on the sizes of the app icons
Figure 4.2.3: User satisfaction on the colors of elements and background
-1
1
3
5
StronglyDisagree
Disagree Neutral Agree StronglyAgree
I thought the sizes of the texts were precize
012345
StronglyDisagree
Disagree Neutral Agree StronglyAgree
I thought the size of the icons used in the app were precise
012345
StronglyDisagree
Disagree Neutral Agree StronglyAgree
I thought the colors of the texts and background were consistent
10
Figure 4.2.4: User satisfaction on the sizes of the buttons
4.3 Test Results on Scenario 2: Alignment, text wrapping and natural flow of app pages
In this scenario, a great user satisfaction was observed. All of the participants reported that the navigation from one page to another, using the slide bar and text wrapping were as good as it was required.
Figure 4.3.1: User satisfaction alignment of text labels and input fields
0
1
2
3
4
5
StronglyDisagree
Disagree Neutral Agree StronglyAgree
I thought the size of the buttons were appropriate
0
1
2
3
4
5
StronglyDisagree
Disagree Neutral Agree StronglyAgree
I thought the texts and inputs fields were properly aligned
11
Figure 4.3.2: User satisfaction on page navigation
Figure 4.3.3: User satisfaction on sliding up and down
Fig: User satisfaction on text wrapping
0
1
2
3
4
5
StronglyDisagree
Disagree Neutral Agree StronglyAgree
I thought I could navigate back and forth quite easily
0
1
2
3
4
5
StronglyDisagree
Disagree Neutral Agree StronglyAgree
I thought I could easily use the sliding bar to slide up and down an app page if necessary
0
1
2
3
4
5
StronglyDisagree
Disagree Neutral Agree StronglyAgree
I thought the texts were wrapped within the input fields
12
4.4 Test Results on Scenario 3: Text input validation, feedback and exceptions
In this test scenario a mixed type of satisfaction was reported by the user. It was due to the fact that most of the input fields were not validated and there had been cases where user was able to enter invalid information in the input fields. For example, when entering a floating point number user could enter number with multiple points, user could enter negative values for volume of different chemical elements, user could submit forms without entering any value in the input fields. Also, when adding multiple volunteers in the last page, the app let the user add a volunteer without filling out the information of the previous volunteer. From a real world application’s perspective, these issues are important to resolve.
Figure 4.4.1: User satisfaction on invalid data entry. Users were not happy since many of the input fields were not validated.
Figure 4.4.2: User satisfaction on feedback. Users thought the feedback was not enough.
0
1
2
3
StronglyDisagree
Disagree Neutral Agree StronglyAgree
I thought I could not enter invalid information
0
1
2
3
4
5
StronglyDisagree
Disagree Neutral Agree StronglyAgree
I thought the app provided reasonable feedback
13
Figure 4.4.3: User satisfaction on application crash
Figure 4.4.4: User satisfaction on useful help information
4.5 Overall User satisfaction
This section reports the overall user satisfaction of the users after using the app. This section is based on the results from the general post-test questionnaire. User satisfaction on many aspects were great. When asked if the users liked the app or not, everyone reported that he/she liked the app.
0
2
4
6
StronglyDisagree
Disagree Neutral Agree StronglyAgree
I thought the app did not crash suddenly
0
1
2
3
4
5
StronglyDisagree
Disagree Neutral Agree StronglyAgree
I thought the app provided useful help information
14
Figure 4.5.1: Users’ understanding level of the app
Figure 4.5.2: Users satisfaction on app latency
Figure 4.5.3: Users’ familiarity with the app labels
0
1
2
3
4
5
StronglyDisagree
Disagree Neutral Agree StronglyAgree
I thought I clearly understood the objective of the app
0
1
2
3
4
5
StronglyDisagree
Disagree Neutral Agree StronglyAgree
I thought the app took precise time to respond to interactions
0
1
2
3
4
5
StronglyDisagree
Disagree Neutral Agree StronglyAgree
I thought I am familiar with the app labels
15
Figure 4.5.4: User satisfaction on sudden crash
Figure 4.5.5: User satisfaction on app complexity.
Figure 4.5.6: User satisfaction on the app interface
0
1
2
3
4
5
StronglyDisagree
Disagree Neutral Agree StronglyAgree
I thought the app did not behave weirdly
0
12
34
5
StronglyDisagree
Disagree Neutral Agree StronglyAgree
I thought the design of the app was too complex
0
1
2
3
4
StronglyDisagree
Disagree Neutral Agree StronglyAgree
I thought I was satisfied with the app interface
16
Figure 4.5.7: Whether user would use the app for volunteering purposes
Figure 4.5.8: Whether users liked the app or not
4.6 User comments
“It was easy to use”
“Sleek design, easy to use, easy to use with phone, fast”
“Going back and forth with navigation bar was simple/easy”
“It was very fast and responsive. It did well to use my native calendar and clock”
“I liked the drop down menu bar in the upper left of the screen”
0
1
2
3
4
5
StronglyDisagree
Disagree Neutral Agree StronglyAgree
If I were a volunteer of Oklahoma Bluthumb project I would certainly use this app
Did you like the app?
Yes No
17
5 Conclusions
Based on the tests conducted the following issues are suggested to resolve
Issue 1: ‘Submit’ button does not work
Description: In the final page of the app after a user finishes entering all the information he enters the ‘Submit’ button. But when using the app there was no way to tell if the button worked. For a practical application when user enters the submit button usually a pop up message is displayed and a new form may also appear. There were no such events in our case.
Resolution: A pop up window can be displayed confirming user that their data has been submitted and a new log file can appear.
Issue 2: Focus on input field does not change on enter
Description: In the test when user was done entering a value for a text input field and pressed enter the cursor did not move to the next input field. The user had to touch outside of the
18
keyboard and then select the next input field again to enter values.
Resolution: The developers need to add some functionality to move the focus on the next field after enter or go is pressed in the keyboard.
Issue 3: Input fields were not validated
Description: Many of the input fields were not properly validated
o Invalid fractions: User could enter fractions with multiple points o Negative values: User could enter negative volumes for chemical elements such as
Oxygen, Nitrogen, Phosphorus etc. o Empty volunteer: The app would let user add multiple volunteers without checking
any information in the previous volunteer form was entered or not.
Resolution:
o Validate form fields as necessary so that invalid fractions cannot be entered o Validate form fields as necessary so that negative volume cannot be entered o Check whether a volunteer information has been added before proceeding to the
next volunteer
19
Undergraduate group member attendance at testing
Test # Date and time Name of the undergrad students
Test 1 3.00 – 4.00 PM, 4/13/18 Zachary Paris
Test 2 4.00 – 5.00 PM, 4/13/18 Zachary Paris
Test 3 5.00 – 6.00 PM, 4/13/18 Zachary Paris
Test 4 3.00 – 4.00 PM, 4/17/18 Nobody attended
Test 5 4.00 – 5.00 PM, 4/17/18 Nobody attended
Test 6 5.00 – 6.00 PM, 4/17/18 Nobody attended
Appendix B
20
BUG REPORT FORM
NUMBER BUG NAME UNIQUENESS LOCATION DESCRIPTION
1 Submit button
worked or not?
Consistent
through all test
Final page No popup message or a
new log form appeared
when pressed
2 User could
enter invalid
fraction
numbers
Form fields that
allow fraction
values
Invalid fractions like
12.32.25 could be
entered
3 User could
enter negative
volumes
Form fields which
required to enter
volumes
Negative volumes like -
12 could be entered
Definitions___________________________________________________________________________________________
NUMBER: A bug number is an identifier provided by the usability test administrator when a user encounters a bug.
BUG NAME: The name of an encountered bug which can easily be distinguishable and comprehendible later on.
UNIQUENESS: An asterisk is assigned if the user is encountering the bug for the first time and no other user has encountered it before.
LOCATION: The page where the bug was encountered.
DESCRIPTION: A formal description of the bug that has been encountered.
Appendix C
21
Computer User Interface Usability Testing Consent Form
You are being invited to participate in a research study to determine the usefulness and usability of computer user interfaces. This study is being conducted by Dr. Robert Pastel of Michigan Technological University Computer Science Department and students in Dr. Pastel's Human-Computer Interaction (MCI) courses. The students are performing the usability tests as part of their project and to fulfill the HCI course requirements.
There are no known risks if you decide to participate in this research study. There are no costs to you for participating in the study. The information you provide and tasks that you will perform will determine the usefulness and usability of user interfaces. The questionnaires and the tasks should take less than an hour to complete. The information collected may not benefit you directly, but the information learned in this study should provide more general benefits.
The questionnaires and test are anonymous. Do not write your name on the survey. No one will be able to identify you or your answers, and no one will know whether or not you participated in the study except for the instructor of the class that is giving you credit for participating, Should the data be published, no individual information will be disclosed.
Your participation in this study is voluntary. By completing the questionnaires and performing the tasks, you are voluntarily agreeing to participate. You are free to decline to answer any particular question you do not wish to answer or not to perform a task for any reason.
If you have any questions about the study, please contact Dr. Robert Pastel, Assistant Professor, Computer Science Department, Michigan Technological University, Houghton, MI 49931.
The MTU Institutional Review Board has reviewed my request to conduct this project. If you have any concerns about your rights in this study, please contact Joanne Polzien of the MTU-IRB at 906-487-2902 or email jpolzien@mtu.edu.
Participant signature and date:
Appendix D
22
PRE-TEST QUESTIONNAIRE (1 of 2)
Participant ID: _______ Age (approximate) : _______ Gender : _
I am currently studying at a university:
□ Yes □ No
If you answered YES, what is your major? _______________________________________
I normally use a smartphone with the following operating system:
o iOS
o Android
o Do not use smartphone
The physical size of my smartphone screen is (only check one):
o 3.7 ~ 4.7 inches o 5.0 ~ 5.2 inches
o > 5.2 inches o Not applicable (check only if you do not use a smartphone)
I have been using a smartphone for (only check one):
o less than a year o 1~2 years
o 2~3 years
o more than 3 years o Not applicable (check only if you do not use a smartphone)
I use my smartphone for the following activities (check all that apply):
□ talking to people □ texting □ emailing □ playing games □ web browsing □ Not applicable (check only if you do not use a smartphone)
Appendix E
23
POST TEST QUESTIONNAIRE (Scenario 1)
Please indicate your level of satisfaction with the app you have just worked on. Fill in the circle that reflects your satisfaction level:
I thought the sizes of the texts were precise
Strongly Disagree ○ ○ ○ ○ ○ Strongly Agree
I thought the size of the icons used in the app were precise
Strongly Disagree ○ ○ ○ ○ ○ Strongly Agree
I thought the colors of the texts and background were consistent
Strongly Disagree ○ ○ ○ ○ ○ Strongly Agree
I thought the size of the buttons were appropriate
Strongly Disagree ○ ○ ○ ○ ○ Strongly Agree
In the space below provide any comment you like to share about this test scenario:
Appendix F
24
POST TEST QUESTIONNAIRE (Scenario 2)
Please indicate your level of satisfaction with the app you have just worked on. Fill in the circle that reflects your satisfaction level:
I thought the texts and field were properly aligned
Strongly Disagree ○ ○ ○ ○ ○ Strongly Agree
I thought I could navigate back and forth quite easily
Strongly Disagree ○ ○ ○ ○ ○ Strongly Agree
I thought I could use the sliding bar to navigate up and down a app page if necessary
Strongly Disagree ○ ○ ○ ○ ○ Strongly Agree
I thought the text were wrapped within the input field
Strongly Disagree ○ ○ ○ ○ ○ Strongly Agree
In the space below provide any comment you like to share about this test scenario:
Appendix G
25
POST TEST QUESTIONNAIRE ( Scenario 3)
Please indicate your level of satisfaction with the app you have just worked on. Fill in the circle that reflects your satisfaction level:
I thought I could not enter invalid information
Strongly Disagree ○ ○ ○ ○ ○ Strongly Agree
I thought the app provided reasonable feedback
Strongly Disagree ○ ○ ○ ○ ○ Strongly Agree
I thought the app did not crash all on a sudden
Strongly Disagree ○ ○ ○ ○ ○ Strongly Agree
I thought the app provided useful help information
Strongly Disagree ○ ○ ○ ○ ○ Strongly Agree
In the space below provide any comment you like to share about this test scenario:
Appendix H
26
POST TEST QUESTIONNAIRE (General)
Please indicate your level of satisfaction with the app you have just worked on. Fill in the circle that reflects your satisfaction level:
I thought I clearly understood the objective of the app
Strongly Disagree ○ ○ ○ ○ ○ Strongly Agree
I thought the app took precise time to respond to interactions
Strongly Disagree ○ ○ ○ ○ ○ Strongly Agree
I thought I am familiar with the app labels
Strongly Disagree ○ ○ ○ ○ ○ Strongly Agree
I thought the app did not behave weirdly
Strongly Disagree ○ ○ ○ ○ ○ Strongly Agree
I thought the design of the app was too complex
Strongly Disagree ○ ○ ○ ○ ○ Strongly Agree
I thought I was satisfied with the app interface
Strongly Disagree ○ ○ ○ ○ ○ Strongly Agree
If I were a volunteer of Oklahama blue thumb project I would certainly use this app
Strongly Disagree ○ ○ ○ ○ ○ Strongly Agree
Based on your experience please provide some feedback on the following questions:
Did you like the app? ___________________
In your opinion, is there anything that you did not like and why?
Is there anything that you liked very much and why?
Please share any comments you have about the study in general:
Recommended