Design and User Evaluation of Augmented-Reality Interfaces

TopicUsability Evaluation Methods

Payel BandyopadhyayMSc Student

University of Helsinki

Outline❏ Introduction❏ Lab based usability testing❏ Procedure ❏ Advantages❏ Disadvantages❏ Questions/Comments❏ References

Introduction❏ Usability Evaluation Methods ❏ Usability testing

❏ Lab studies❏ Field studies

❏ enables us to quantify the extent to which a product meets the needs of its intended users [3]❏ “performance measurement to determine whether usability goals are achieved”

[1]

Figure 1: Usability testing evaluation methods [6]

Laboratory based usability testing (1/2)

❏ Conducted in laboratories with test ❏ Not necessary to take place in dedicated “laboratory” [9]❏ Controlled environment❏ Can be conducted in various controlled environments

❏ office [9] ❏ hall way [9] ❏ simulator [9] ❏ others

❏ Mimic real life scenario❏ “The design of a laboratory usability test summarizes the issues of concern, the tasks to be observed, the

questions to ask, and the criteria for screening the people who participate. The researcher creates and follows a detailed session script so that all participants receive the same instructions and error remediation, while performing the agreed-upon tasks using the same data” [4]

Figure : A laboratory set up for conducting usability testing [1]

Laboratory based usability testing (2/2) ❏ Evaluate the effectiveness and ease of use of your prototype/product [2]❏ Gain unique insights

❏ users' needs ❏ preferences ❏ observing their behavior as they perform typical activities [4]

❏ Experimenter has control of ❏ assignments of subjects ❏ treatment variables ❏ manipulation of variables

❏ Exploratory testing [4]❏ Qualitative data

❏ video/audio recordings of user activity❏ interviews❏ feedback from users

❏ Performance Testing [4]❏ Quantitative data

❏ statistics/numbers❏ logs/ diary

Procedure (1/2)❏ Lab environment should be as real as possible [10]❏ Ensure that your prototype works exactly the way you want❏ Define your research goal❏ Recruit participants match your

❏ “user task” goal❏ research goal

❏ Pre-screen participants (if applicable)❏ questionnaires❏ interviews❏ diary

❏ Participant(s) performs a pre-defined set of tasks ❏ Explain them the exact “user task” to perform❏ Show a demo

Procedure (2/2)

❏ Behave exactly the same way with all users ❏ no biasing

❏ Inform users (if applicable)❏ video recordings❏ audio recordings❏ data logs

❏ After experiment provide participants (if applicable)❏ questionnaires❏ interview❏ diary

❏ Reward the participants

Figure 3: A sample of how video and audio recordings can be done to evaluate your prototype/subject [8]

Figure 3: A sample user task definition [7]

Figure 3: A sample user task definition [1]

Advantages ❏ Opportunity to focus

❏ specific phenomena of interest [1]❏ High-quality data

❏ video recording ❏ of the display ❏ user interaction

❏ All users experience the same setting ❏ higher quality data [5]

❏ The difficulties in conducting and collecting data are significantly reduced ❏ as compared to field based usability tests [1]

❏ Highly replicable (of real world) [1]❏ Facilitate good data collection [1]

Disadvantages

❏ Less reliable data ❏ “real life” users

❏ Cannot address factors and issues ❏ occur at the field [1]

❏ Recordings might distract/aware the user❏ Please the experimenter❏ Percentage of real life scenario simulation

❏ critical factor in terms of the validity of the usability test [1]

Figure 2: A comparison of set-up of usability testing in real world and lab environment. This figure shows a difference between computer setting in pre-school (left) and USE lab (right) [5]

Figure: Difference in final results in field based usability testing and lab based usability testing [1]

Figure: Difference in user behaviour in field based usability testing and lab based usability testing [1]

Questions/Comments

References (1/2)[1] Henry Been-Lirn Duh, Gerald C. B. Tan, and Vivian Hsueh-hua Chen. 2006. Usability evaluation for mobile device: a comparison of laboratory and field tests. In Proceedings of the 8th conference on Human-computer interaction with mobile devices and services (MobileHCI '06). ACM, New York, NY, USA, 181-186. DOI=10.1145/1152215.1152254 http://doi.acm.org/10.1145/1152215.1152254

[2] Brenda Battleson, Austin Booth, Jane Weintrop, Usability testing of an academic library Web site: a case study, The Journal of Academic Librarianship, Volume 27, Issue 3, May 2001, Pages 188-198, ISSN 0099-1333, http://dx.doi.org/10.1016/S0099-1333(01)00180-X.(http://www.sciencedirect.com/science/article/pii/S009913330100180X)

[3] Rosenbaum, S., "Usability evaluations versus usability testing: when and why?," Professional Communication, IEEE Transactions on , vol.32, no.4, pp.210,216, Dec 1989, doi: 10.1109/47.44533

URL: http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=44533&isnumber=1691

[4] Rosenbaum, S.; Kantner, L., "Field Usability Testing: Method, Not Compromise," Professional Communication Conference, 2007. IPCC 2007. IEEE International , vol., no., pp.1,7, 1-3 Oct. 2007, doi: 10.1109/IPCC.2007.4464060

URL: http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4464060&isnumber=4464038

[5] Razak, F.H.A.; Hafit, H.; Sedi, N.; Zubaidi, N.A.; Haron, H., "Usability testing with children: Laboratory vs field studies," User Science and Engineering (i-USEr), 2010 International Conference on , vol., no., pp.104,109, 13-15 Dec. 2010, doi: 10.1109/IUSER.2010.5716733

URL: http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=5716733&isnumber=5716710

References (2/2)[6] Alshamari, M.; Mayhew, P., "Task Design: Its Impact on Usability Testing," Internet and Web Applications and Services, 2008. ICIW '08. Third International Conference on , vol., no., pp.583,589, 8-13 June 2008, doi: 10.1109/ICIW.2008.20

URL: http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4545676&isnumber=4545574

[7] Morten Sieker Andreasen, Henrik Villemann Nielsen, Simon Ormholt Schrøder, and Jan Stage. 2007. What happened to remote usability testing?: an empirical study of three methods. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '07). ACM, New York, NY, USA, 1405-1414. DOI=10.1145/1240624.1240838 http://doi.acm.org/10.1145/1240624.1240838

[8] Book Section D 2004@ 978-3-540-23086-1, Mobile Human-Computer Interaction - MobileHCI 2004. V 3160, Lecture Notes in Computer Science, Brewster, Stephen, Dunlop, Mark, 10.1007/978-3-540-28637-0_6T, Is It Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the Field, http://dx.doi.org/10.1007/978-3-540-28637-0_6, Springer Berlin Heidelberg, 2004-01-01, Kjeldskov, Jesper, Skov, MikaelB. Als, BenedikteS. Høegh, RuneT. pp. 61-73

[9] A Review of MobileHCI Research Methods. / Kjeldskov, Jesper; Graham, C., Proceedings of the 5th International Mobile HCI 2003 conference, September 8-11 2003, Udine, Italy: Lecture Notes in Computer Science. Springer, 2003.

[10] http://msdn.microsoft.com/en-us/library/windows/desktop/aa468538.aspx