Usability Studies with JapaneseChildren for Mobile Applications:
Conducting a Meta Analysis
Master‘s Thesis DefenceBernd Hollerit
Supervisor: Wolfgang SlanyExaminers: Keith Andrews, Andreas Holzinger
Bernd Hollerit 2
Schedule
• Motivation• Catrobat• Usability• Definition• Importance• Evaluation Techniques• Our Approach
• Catrobat Usability Evaluation• Evaluation of Surveys
• Identified Usability Problems• Catrobat Redesign• Paintroid Redesign• Suggestions for Japanese Market
• Conclusion• Outlook• Q&A
2013-11-28
Bernd Hollerit 3
Motivation
• IT important for most businesses• Not only IT staff need knowledge, but everyone working in an office• Most thought after IT skills nowadays are programming and app
development• IT sector is growing and is expected to keep thriving• 61% of IT executives announced plans to hire web- & app developers• Too few women in IT• Cell phone usage among children on the rise
2013-11-28
Bernd Hollerit 4
Catrobat
• Visual programming language• Inspired by Scratch• Developed for smartphones and
tablets• Targets children and teenagers• Goal: learn mathematical and
computational ideas, think creatively, reason systematically, and work collaboratively
2013-11-28
Bernd Hollerit 5
Usability – Definition
1. Effectiveness2. Learnability3. Efficiency4. Memorability5. Errors6. Satisfaction
2013-11-28
Bernd Hollerit 6
Usability – Importance
• If an application is difficult to use…• If the app fails to clearly convey what it has to offer…• If users get lost while navigating through the app…• If the purpose of an app is hard to figure out…• If the app doesn‘t fulfill a user‘s expectations…
2013-11-28
Bernd Hollerit 7
Usability – Importance
• If an application is difficult to use…• If the app fails to clearly convey what it has to offer…• If users get lost while navigating through the app…• If the purpose of an app is hard to figure out…• If the app doesn‘t fulfill a user‘s expectations…
…people will stop using it
• Usability is a necessary condition for survival
2013-11-28
Bernd Hollerit 8
Usability – Evaluation Techniques
• Early Focus on Users and Tasks• Empirical Measurement• Iterative Design• Parallel Design• GOMS (Goals, Operators,
Methods, Selection Rules)• Human Processor Model• Keystroke Level Modeling• Card Sorts
• Tree Tests• Ethnography• Heuristic Evaluation• Usability Inspection• Pluralistic Inspection• Consistency Inspection• Activity Analysis• Task Analysis
2013-11-28
Bernd Hollerit 9
Usability – Evaluation Techniques (cont.)• Focus Groups• Surveys/Questionnaires• Remote Usability Testing• Remote Usability Testing for
Mobile Devices• Thinking Aloud• RITE (Rapid Iterative Testing and
Evaluation) Method• Subjects-in-Tandem or Co-
Discovery
• Component-Based Usability Testing• Rapid Prototyping• Cognitive Walkthrough• Benchmarking• Persona• Meta-Analysis
2013-11-28
Bernd Hollerit 10
Usability – Our Approach
• Meta Analysis of a Focus Group, with Surveys/Questionnaires, Activity Analysis, Thinking Aloud, Cognitive Walkthrough and, to a lesser extent, GOMS and Task Analysis• Focus Group: Japanese teenagers• Survey before hands-on experience, questionnaire after• Start with Thinking Aloud• If children can‘t be encouraged to talk -> Activity Analysis, Cognitive
Walkthrough, GOMS, Task Analysis
2013-11-28
Bernd Hollerit 11
Catrobat Usability Evaluation
• 4 children aged 13-17 years• Tests lasted between 45 and 99 minutes• Facilitator creates timestamped log of activities
1. Survey before actual test2. Hands-on experience of the Catrobat software3. Questionnaire after using Catrobat
2013-11-28
Bernd Hollerit 12
Evaluation of Surveys
• Children agreed or strongly agreed that the usage of Catrobat is fun, interesting, cool and entertaining.• They disagreed or strongly
disagreed that Catrobat is frustrating and boring.• Half of the children agreed that
Catrobat was confusing, with the other half disagreeing.
• „Would you be interested in taking a computer science class in high school?”
2013-11-28
Before Catrobat Usage After Catrobat Usage
Maybe yes, maybe no Probably yes
Probably yes Definitely yes
Probably yes Probably yes
Maybe yes, maybe no Probably yes
Bernd Hollerit 13
Identified Usability Problems
• Children tap field for number input when they want to move a brick around• Children draw outside of boundaries of
Paintroid image• Children don't know the Paintroid tool “magic”• Children don't know the Paintroid tool “stamp”• Children exit Paintroid without saving• Images open on a very tiny zoom level• Localisation errors in Paintroid• Long-press actions aren't intuitive
• Interface is generally confusing• Inconsistent „Add“-
Buttons
-> Uniform Redesign
2013-11-28
Bernd Hollerit 14
Catrobat Redesign
2013-11-28
Bernd Hollerit 15
Catrobat Redesign
2013-11-28
Bernd Hollerit 16
Catrobat Redesign
2013-11-28
Bernd Hollerit 17
Catrobat Redesign
2013-11-28
Bernd Hollerit 18
Catrobat Redesign
2013-11-28
Bernd Hollerit 19
Catrobat Redesign
2013-11-28
Bernd Hollerit 20
Catrobat Redesign
2013-11-28
Bernd Hollerit 21
Paintroid Redesign
2013-11-28
Bernd Hollerit 22
Suggestions for Japanese Market
• Focus on features regarding drawing and animation• E.g. with templates, pre-set
images
• Localisation not complete, fixed missing translations• One younger participant (13
years old) was unfamiliar with pinch-to-zoom
2013-11-28
Bernd Hollerit 23
Conclusion
• Prepared, conducted and reviewed usability evaluation in order to improve the user experience of Catrobat• Performed Meta-Analysis consisting of Questionnaires/Surveys,
Activity Analysis, Cognitive Walkthrough, Thinking Aloud, Task Analysis and a Consistency Inspection with Japanese children• Suggested improvements and discussed resulting changes• Received Austrian national award for innovation for multimedia and
e-business• Promotion from Google in the Google Play Store (Education Section)
2013-11-28
Bernd Hollerit 24
Outlook
• Reevaluate the redesign• Assess actual user opinions• http://catrobat.org – Developer website hosting the latest builds of
Pocket Code and Pocket Paint• https://pocketcode.org – Hosts user-created apps that can be
downloaded and run by any Pocket Code version • http://blog.catroid.org – Catrobat Blog• https://github.com/Catrobat – Catrobat GitHub Project
2013-11-28
Thank you for your attention!
Q&A
Bernd Hollerit 26
Sources
• [ADAMS2009] Adams, Stephen. Children get first mobile phone at average age of eight. 2009 http://www.telegraph.co.uk/technology/news/4680507/Children-get-first-mobile-phone-at-average-age-of-eight.html
• [ANDREWS2013] Andrews, Keith. Human-Computer Interaction -- Course Notes. 2013 http://courses.iicm.tugraz.at/hci/hci.pdf• [BACON-SHONE2009] Bacon-Shone, John and Hui, Desmond. Design and construction of the Hong Kong Creativity Index.
Publications Office of the European Union. p. 91--102 2009 http://ec.europa.eu/education/lifelong-learning-policy/doc/creativity/report/hongkong.pdf
• [BURTSCHER2012] Burtscher, Daniel. Introduction of a Continuous Integration Process in an Open Source Project. 2012• [CARD1983] Card, Stuart. The psychology of human-computer interaction. L. Erlbaum Associates. 1983• [CARD1986] Card, S.K and Moran, T. P. and Newell, A.. Handbook of Perception and Human Performance. John Wiley and sons. p.
1-35 1986• [CROPLEY2000] Cropley, Arthur. Defining and measuring creativity: Are creativity tests worth using?. Routledge. p. 72--79 2000
http://www.informaworld.com/openurl?genre=article&doi=10.1080/02783190009554069&magic=crossref• [CSIKSZENTMIHALYI2009] Csikszentmihalyi, M.. Creativity: Flow and the Psychology of Discovery and. HarperCollins. 2009
http://books.google.co.jp/books?id=aci_Ea4c6woC• [DAVIDSON2012] Davidson, Danica. Manga grows in the heart of Europe. 2012 http://geekout.blogs.cnn.com/2012/01/26/manga-
in-the-heart-of-europe/• [FIDGEON2011] Fidgeon, Tim. Ethnography – when and how to use. 2011 http://www.spotlessinteractive.com/articles/usability-
research/ethnography-when-and-how-to-use.php
2013-11-28
Bernd Hollerit 27
Sources
• [FISHER2004] Fisher, R. and Williams, M.. Unlocking Creativity: A Teacher's Guide to Creativity Across the Curriculum. Taylor & Francis. 2004 http://books.google.co.jp/books?id=W6WOTKVVjCwC
• [FRANKEN2006] Franken, R.E.. Human Motivation. Thomson/Wadsworth. 2006 http://books.google.at/books?id=hvW6AAAACAAJ• [GARRETT2003] Garrett, Jesse James. The Elements of User Experience. David Dwyer. 2003• [GOULD1985] Gould, John D. and Lewis, Clayton. Designing for Usability: Key Principles and What Designers Think. Communications of
the ACM. 1985 http://dl.acm.org/citation.cfm?id=3170• [GREENSPAN2011] Greenspan, Daniel. Top IT Skills In Demand for 2012. 2011
http://www.itcareerfinder.com/brain-food/blog/entry/top-9-it-skills-in-demand-for-2012.html• [GRITSCHACHER2011] Gritschacher, Tobias. A Community Website for Interactive Mobile Content Created by Children and Teenagers.
2011• [GTB2008] Global Telecoms Business. A century of influence. 2008
http://www.globaltelecomsbusiness.com/Article/2199338/Search/Results/A-century-of-influence.html• [HOLLANDERS2009] Hollanders, Hugo and Van Cruysen, Adriana. Design and construction of the Hong Kong Creativity Index.
Publications Office of the European Union. p. 41--81 2009 http://ec.europa.eu/education/lifelong-learning-policy/doc/creativity/report/design.pdf
• [ISO1998] International Organization for Standardization. ISO 9241-11:1998 Ergonomic requirements for office work with visual display terminals (VDTs) -- Part 11: Guidance on usability. 1998 http://www.iso.org/iso/home/store/catalogue_tc/catalogue_detail.htm?csnumber=16883
• [ISO2013] International Organization for Standardization. About ISO. 2013 http://www.iso.org/iso/about/about
2013-11-28
Bernd Hollerit 28
Sources
• [JASTRZEMBSKI2007] Jastrzembski, Tiffany S. and Charness, Neil. The Model Human Processor and the Older Adult: Parameter Estimation and Validation Within a Mobile Phone Task. Journal of Experimental Psychology: American Psychological Association. 2007
• [JOHNSON2013] Johnson, Daniel and Jones, Christian. Beyond the beat-em-up: video games are good for young people. 2013 http://pda.sciencealert.com.au/features/20133008-24746-2.html
• [KELLEHER2006] Kelleher, Caitlin. Motivating Programming: using storytelling to make computer programming attractive to middle school girls. 2006
• [KIERAS1993] Kieras, David. Using the Keystroke-Level Model to Estimate Execution Times. University of Michigan. 1993 http://www.pitt.edu/~cmlewis/KSM.pdf
• [KNAPITSCH2012] Knapitsch-Scarpatetti-Unterwegen, Ferdinand R.. Usability Testing of Mobile Applications for Children. 2012• [KRUG2000] Krug, Steve. Don't Make Me Think: A Common Sense Approach to Web Usability. Berkeley, CA: New Riders. 2000• [MARSHALL2009] Marshall, Andrew. Importance of IT in business. 2009 http://ezinearticles.com/?Importance-of-IT-in-
Business&id=3221599• [MAYER2012] Mayer, Stefan. Testen von mobilen HTML5 Web-Applikationen. 2012• [MEDLOCK2002] Medlock, Michael C. and Wixon, Dennis and Terrano, Mark and Romero, R. and Fulton, Bill. Using the RITE
method to improve products: A definition and a case study. 2002• [NIELSEN1995] Nielsen, Jakob. How to Conduct a Heuristic Evaluation. 1995 http://www.nngroup.com/articles/how-to-conduct-a-
heuristic-evaluation/
2013-11-28
Bernd Hollerit 29
Sources
• [NIELSEN1996] Nielsen, Jakob. Improving System Usability Through Parallel Design. 1996 http://www.nngroup.com/articles/parallel-design/
• [NIELSEN2000] Nielsen, Jakob. Why You Only Need to Test with 5 Users. 2000 http://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/
• [NIELSEN2001] Nielsen, Jakob and Snyder, Carolyn and Molich, Rolf and Farrell, Susan. E-commerce user experience. Nielsen Norman Group. 2001
• [NIELSEN2012A] Nielsen, Jakob. Usability 101: Introduction to Usability. 2012 http://www.nngroup.com/articles/usability-101-introduction-to-usability/
• [NIELSEN2012B] Nielsen, Jakob. Thinking Aloud: The #1 Usability Tool. 2012 http://www.nngroup.com/articles/thinking-aloud-the-1-usability-tool/
• [NYT1998] Richtel, Matt. Making web sites more `usable' is former Sun-engineer's goal. 1998 http://www.nytimes.com/library/tech/98/07/cyber/articles/13usability.html
• [RESNICK2013] Resnick, Mitch. Mitch Resnick: Let's teach kids to code. 2013 http://www.ted.com/talks/mitch_resnick_let_s_teach_kids_to_code.html
• [RUNCO2010] Runco, M.A.. Creativity: Theories and Themes: Research, Development, and Practice. Elsevier Science. 2010 http://books.google.at/books?id=nJlfKE-meqEC
2013-11-28
Bernd Hollerit 30
Sources
• [S2011] buzzle.com. Importance of Information Technology. 2011 http://www.buzzle.com/articles/importance-of-information-technology.html
• [SAWYER2006] Sawyer, R. Keith. Explaining creativity: The science of human innovation. Oxford University Press. 2006 http://ukcatalogue.oup.com/product/9780195304459.do?keyword=keith+sawyer&sortby=bestMatches
• [SCHELHOWE2012] . IDC '12: Proceedings of the 11th International Conference on Interaction Design and Children. ACM. 2012• [SCRATCH2013] Massachusetts Institute of Technology. Scratch. 2013 http://scratch.mit.edu/• [SPENCER2003] Spencer, Donna. Card-Based Classification Evaluation. 2003 http://boxesandarrows.com/card-based-
classification-evaluation/• [SPENCER2009] Spencer, D. and Garrett, J.J.. Card Sorting: Designing Usable Categories. Rosenfeld Media. 2009
http://books.google.at/books?id=_h4D9gqi5tsC• [STERNBERG1999] Sternberg, R.J.. Handbook of Creativity. Cambridge University Press. 1999 http://books.google.ca/books?
id=d1KTEQpQ6vsC• [STIERNA2009] Stierna, Johan and Villalba, Ernesto. Design and construction of the Hong Kong Creativity Index. Publications
Office of the European Union. p. 103--111 2009 http://ec.europa.eu/education/lifelong-learning-policy/doc/creativity/report/science.pdf
• [TED2006] TED Conferences, LLC. TED Talks: Ken Robinson says schools kill creativity. 2006 http://www.ted.com/talks/lang/en/ken_robinson_says_schools_kill_creativity.html
• [TED2013] TED Conferences, LLC. TEDx Events. 2013 https://www.ted.com/tedx
2013-11-28
Bernd Hollerit 31
Sources
• [TREITLER2012] Treitler, Peter. A generic approach towards smart self-rendering mobile user interfaces. 2012
• [TURKLE2011] Turkle, Sherry. Alone Together: Why We Expect More from Technology and Less from Each Other. Basic Books. 2011 http://books.google.at/books?id=_Dhf5xEZZD0C
• [UNITID2013] UNITiD and contributors. Android Patterns. 2013 http://www.androidpatterns.com/uap_pattern/pinch-spread
• [V2010] phonearena.com. No surprise—more children owning cell phones. 2010 http://www.phonearena.com/news/No-surprise—more-children-owning-cell-phones_id8622
• [VILLALBA2009] Villalba, E.. Measuring Creativity: Proceedings for the Conference ``Can Creativity be Measured?". Brussels, May 28-29, 2009. Publications Office of the European Union. 2009 http://books.google.at/books?id=DczbcQAACAAJ
• [WEBDESIGNERDEPOT2009] Apai, Walter. Interview with web usability guru, Jakob Nielsen. 2009 http://www.webdesignerdepot.com/2009/09/interview-with-web-usability-guru-jakob-nielsen/
• [WEISBERG1993] Weisberg, R.W.. Creativity: Beyond the Myth of Genius. Freeman. 1993 http://books.google.at/books?id=_VNwQgAACAAJ
• [WEISS2002] Weiss, Scott. Handheld usability. John Wiley and Sons, Ltd. 2002
2013-11-28