Usability in the Wild: Mindset and Framework

Usability Testing in the Wild : Mindset and Framework

David WalczykClient: Columbia University, NYC

situation & myths/

precepts & methods

situation & myths/

precepts & methods

situation & myths/

precepts & methods

OLD MYTHSinformation is...

scarce, linear, concrete, analog, surface, closed, non-visual, non-spatial, discrete,

hierarchical, mutually exclusive, phonetic, etc...

NEW MYTHSinformation is...

abundant, holistic, non-linear, deep, abstract, computational, executed, exploratory, behavioral, symbolic,

recursive, networked, emergent, etc...

situation & myths/

precepts & methods

when media change people change.

new media are extensions & amputations.

new media create new environments.

preconceived figures lead tounperceived grounds.

old methods confirm old ways and conceal new ones.

insight and understanding trumps a point of view.

situation & myths/

precepts & methods

MethodsFirst Thinking and Then Doing

BFI Personas and Scenarios 1

Steve the Student Heather the Hipster Tom the Teacher Adam the Architect Helen the Hippie

Steve isn't looking for much exploration here. He wants a clear path to the information sought so he can complete his homework.

Heather is going to want to explore the site. Her major concerns are going to be finding biographical information and how to get involved.

Tom's primary focus is going to be collecting ideas for his class – activities, lesson plans, audio/video materials. A secondary concern is going to be networking with other professionals to learn about what worked for them.

Adam will be interested in networking/exchanges with like minded designers and their projects.

Helen will be on BFI to read Design Science News and to listen and watch lectures.

Searching for biographical information about Buckminster Fuller from an authoritative source.

Heather is visiting the site to learn more about Buckminster Fuller.

Tom is looking for curriculum ideas and maybe some insight/success stories from others in his field.

Adam is looking for networking opportunities on the site.

Helen is looking for the new issue of Design Science News.

Grew up with WWW. Very tech savvy . Very comfortable. Very comfortable. Can get by.

Steve is coming to the site for information about Bucky the man.

Heather is coming to the site for information about Bucky the man, but is going to explore: community content, current events, ways to get involved, etc.

Tom is coming to the site from a sustainability perspective. He's going to need education-specific resources and community content.

Adam will come to the site to brush up on Buckminster Fuller the man, network with like minded professionals, and keep abreast of sustainability news and events.

Helen is looking for current news and updates to the archive.

Steve is going to expect to find familiar navigation mechanisms, like dynamic menus, that lead him to the information sought.

Heather is going to expect clear navigation and web 2.0 features.

Tom is going to expect logical navigation, and grade-specific educational resources.

Adam is going to expect logical navigation, biographical and historical resources, an easy registration process, and community content.

Helen is going to expect things to be where she’s used to finding them. The structure of the site may not make sense, but she has learned how to navigate it.


Motivations Scenarios Site Features Behaviors on Current BFI Website Behaviors on Redesigned BFI Website

To learn more about Bucky and his connection to the sustainability movement

Read about geodesic domes and thought Bucky sounded familiar, so she Googled "Buckminster Fuller Dome" and landed on a bfi.org content page about domes.

• Clear and logical navigation.

Googles domes and finds what appears to be a landing page for articles about domes at BFI. However navigation out of that page to other parts of the site is unclear.

Lands on content page from Google. Skims it. Realizes she wants to check out more. Using bread crumbs she goes up a level to see what other content may be of interest to her.

Wants to get involved locally

Inspired by her reading the other day, she decides she wants to contribute locally. She goes back to the BFI homepage to find out about local chapters.

• Clear and logical navigation.

• Easy and secure registration process (doesn't ask for too much or too little info).

• Clear opt-out options.

Goes to BFI homepage via her delicious.com bookmarks. Clicks on the Community Content button, but is unsure as it also reads "Connect with our network of designers." Clicks on the "Use our forums" page and is taken to a How to use forums page. Clicks the back button. Sees a link in the red box in the upper right corner labelled "BFI Forums". Clicks that and is taken to the forums.

Goes to BFI homepage via her delicious.com bookmarks. Selects the "Local Chapters" button. Taken to local chapter landing page where she picks a chapter in her local region. Is excited by all the evens listed on the homepage, and wants to participate. Clicks the "register" link at the top right corner (where it's usually found) to sign up for the email newsletter and to join the forum.

Wants to keep abreast of any events and news from BFI.

Gets too much email and newsletters end up spam. Worried about missing an event at her local BFI chapter.

• RSS/Atom (customizable feeds).

Goes to the BFI website and clicks on the RSS link. She's unsure exactly what content she is subscribing to. She clicks "More options (experts only)" and finds an overwhelming list of potential RSS subscription options. She goes back to the RSS page and subscribes to the basic feed.

Heather realizes she doesn't want to fill up her inbox with emails, so she goes to the BFI homepage and clicks the RSS link in the upper right corner. She is taken to the RSS page where she chooses what content she wishes to subscribe to in her feed reader.

HEATHERThe Hipster"Isn't he the dome guy?"

Demographics

Age: 24Education: liberal arts BA, thinking about grad school

income: <$30,000Bucky familiarity: Vague

Comfort with Technology

Computers: Native.Web: Native.Social web: Belongs to all and a few you haven't head of.

Needs

• Biographical/historical information

• Current events• Community content• Information on

getting involved• Clear and logical

navigationPersonal Background

Heather is tech savvy and hyper-connected like most of her lot. She's into sustainability & is coming to the site to learn more about the man and his connection to the broader sustainability movement, though she hasn't really heard of him beyond being "the dome guy". After exploring the site, Heather is going to want to organize a community chapter. She Googled "Buckminster Fuller Dome" to get to the site.


STEVEThe Student"I don't know, my teacher justassigned it."

Demographics

Age: 16Education: High school sophomoreIncome: None (parents >$90,000)Bucky familiarity: None

Comfort with Technology

Computers: Grew up with them.Web: Doesn’t know a world without it.Social web: Facebook is lame now that Mom’s joined.

Needs

• Access historical information about Bucky in order to complete an assignment

Personal Background

Steve is a 16-year-old high school student finishing his sophomore year. He is very comfortable with online environments and has the latest video game consoles. He has no familiarity with Bucky, and was assigned to write a paper about him for a class. He doesn't read web pages thoroughly – he's more of a page scanner and usually clicks the first link. If he doesn't see something quickly he gives up. He is frustrated that he can't just use Wikipedia for his assignment.

Motivations Scenarios Site Features Behaviors on Current BFI Website Behaviors on Redesigned BFI Website

Learn historical information about Buckminster Fuller

He couldn't use Wikipedia as a source. Clicked on BFI.org as an external link from Bucky article.

• Clear path to biographical information.

Clicks the "Who is Buckminster Fuller?" button on the BFI homepage. Scrolls down to "Basic Biography" clicks link -- finds a timeline. Clicks the back button. Clicks on the "Introduction to Buckminster Fuller" article. Is frustrated that the article has some biographical information but it is not complete.

Steve clicks on the About Buckminster tab and is taken to a landing page that includes a Bucky biography. Links to more in-depth resources via the Education page are included.


frequentinteractions

infrequentinteractions

complexinteractions

simpleinteractions

Stev

e th

e St

uden

t

Adam th

e Arch

itect

Helen th

e Hippie

Heather the Hipster

Tom th

e Teac

her

Usage of current BFI website Usage of redesigned BFI website

Key

Change in website usage

BFI Comparative Analysis : Detailed Evaluations : Home Page 5

Make Grist Space Collective Surfrider Foundation Usability Professionals Association

http://www.makezine.com/ http://www.grist.org/ http://spacecollective.org/ http://www.surfrider.org/ http://www.upassoc.org/From the consistent browser title, site title, and subtitle, it is fairly clear that Make covers do-it-yourself technology projects. Ad placement unfortunately dilutes the message somewhat – banner and Google ads loom above the global navigation – and a large ad for McDonald's does not instill initial trust in the site. But global navigation follows Web standards, counteracting the profusion of ads: a logo home button is on the upper left and utilities such as search and RSS are on the upper right, with a row of multicolored labels across the page presenting navigation options to the six main sections of the website beneath. Labels such as "Blog," "Videos/ Podcasts," and "Forum/Community" signal that Make is a participatory site. The labels are highlighted when moused over, and remain so when selected, acting as "breadcrumbs" to orient the user. Other links are not visually uniform: some are indicated by boxes and others merely with Web standard underlined text. An option to "Suggest a Site!" does not specify why, and simple footer navigation is preceded by yet more Google ads. But the home page's large featured content area reinforces Make's message, flashing features from the site and magazine, as well as recent blog posts. The design in general is not particularly slick, which reinforces the you-can-do-it, craft-oriented nature of the site.

Browser title, site ID, and subtitle make clear that the website provides "environmental news and commentary." But a large banner ad is placed at top of the page, where many users expect to find the main navigation. Instead, main navigation is located on the left. Site hierarchy is displayed through the use of section titles, but section titles themselves are not clickable and the list is so long that the user cannot scan what amounts to a summary of the site without scrolling. Navigation labels such as "News/Analysis" communicate a news-oriented site, but other labels, such as "Victual Reality" impart a kind of "hipness." Although the latter type of labels are opaque to the uninitiated, mouseovers yield some clarification. A clearly labeled toolbox links to useful site utilities, and is located where most users would expect it, at top right. The rest of the page follows a layout fairly typical to news organizations on the Web: several prominently featured images/videos plus other areas corresponding to global navigation categories containing titles and links to the latest stories. The right side of the screen contains more featured content, both internal and external to the site, including a section for "this week's most popular" items, as is typical for news sites. Footer navigation continues the problematic global navigation labels, unfortunately.

An initial Flash animation is difficult to read and potentially annoying to users without the plug-in or who are just impatient. A large text block about the Space Collective text then loads first in the middle of the page, signaling that this site is "different." The text outlines the purpose, but not the nature, of the site, and moreover does so in an manner that reinforces the site's arty/science-fiction identity: users are "terrestrials." Navigation is difficult to pick out, as dark gray elements do not contrast with a black background (though some elements pop out in red on mouseovers). A login utility is in a Web standard upper right location, but a barely visible hover label states that membership is by invitation only. Navigation labeling is also an issue, both in design (small and low contrast) and vocabulary (what is a "timecapsule"?). However, the blog/channel, multimedia nature of the site is immediately implied by the label "recent posts" and the obvious image gallery layout. Also, higher contrast large labels effectively separate page areas and lead the eye down and across the page. Space Collective's invitation-only, members-only policy, and its use of a black background, slick layout, and arcane vocabulary convey "coolness." The site is deliberately less than welcoming, drawing users in by creating a desire to be "in" the group and in the know.

Evocative header images are background to the foundation's logo and illustrate its motto, "Protecting Our Beaches For Over 20 Years." Throughout the page, the use of blue and gray evokes the ocean and the foundation's focus. Global navigation in a Web-standard, top of the screen location uses clear, action-oriented labels to orient the user and convey site contents. The site's appeal for user action continues with other command labeling: "Stay Informed," "Join Surfrider," "Support Surfrider." The center of the page features a slide show that outlines the foundation's strategic initiatives and links to the "What We Do" area of the site, and is followed by links to current campaigns and featured content. On the left are site utilities such as an email newsletter signup box, a search box, and membership/chapter links. Membership links are also grouped in and beneath a color-coded portion of the top menu on the right. Beneath this second membership area are links to the foundation's blogs, podcasts, and other publications, coastal news items, and two vaguely labeled but visually appealing links to "The Ring" and "SocialVibe." Overall, the site's clear organization and layout, imperative labeling, and judicious use of images communicates the foundation's proactive nature, efficient operation, and seriousness of purpose.

As befits a website for a professional association of usability specialists, no initial ads detract from the clarity of a Web standard top left logo and subsequent tabbed global navigation. Navigation labels use recognized syntax, such as "About UPA," so that no mouseovers are required to understand them. Most other links are either standard underlined blue text or clickable graphics. Text about the organization is prominently placed in the center just below the global navigation, and is followed by content that is current and useful to the association's membership: conferences, events, and news. A short section then follows which introduces usability concepts to potential clients. Site utilities, including search, most visited links, and membership registration/log-in are offered at the left and right sides of the screen. Above UPA's contact information and privacy policy at page bottom are what appear to be ads in the form of logos/links to UPA's corporate supporter websites, but these logos establish UPA's credibility by announcing its acceptance by these organizations. Clear labeling, standard placement of page elements, and an unremarkable, business-friendly color scheme convey a sober dedication to usability principles.

BFI Comparative Analysis : Summary Evaluations (1 = poor, 5 = 11

Make Grist Space Collective Surfrider Foundation Usability Professionals Association

Home PageDesign

Interior Page Design

Search Functionality

Features Offered

Navigation

Interaction Design

4.04.53.54.03.5

1.5

5.04.04.54.0

2.51.01.0

4.53.5

4.54.02.0

4.55.0

3.05.0

3.04.04.0

1.01.5

4.04.05.0

BFI Comparative Analysis : SWOT Analysis for BFI.org 4

Strengths Weaknesses Opportunities Threats

• BFI has rich content addressing a wide range of potential uses• Buckminster Fuller and his

ideas and design could be enormously appealing for educational uses• BFI already has a dedicated

community of fans of Buckminster Fuller the man• The BFI Challenge has

created enormous goodwill and exposure for the organization• The Idea Index is unique

and could be leveraged to invite more discussion and interaction

• The BFI website does not fully promote R. Buckminster Fuller's legacy by making his life, designs, and ideas easily understandable by a wide audience• Information on the BFI

website is not organized by user-oriented topics• Interactive functions are not

clearly described or distinguished from one another so not all users who might be interested in participating are reached• Separate registrations are

required for BFI and the BFI Challenge• Top level navigation is

difficult to find, so users may be confused about how to navigate the site• Labels use specialized

vocabulary that new users may not understand• Tagging functionality has

not been fully exploited to enable exploration and discovery of site content• The use of the blog format

results in posts and comments that appear to be orphaned and not actively moderated: the site appears to be a ghost town

• Sites such as Surfrider Foundation have not yet fully developed interactive functionality to encourage participation and a large potential user base may still remain to be captivated/captured• Sites such as Space

Collective and Surfrider Foundation provide poor search capabilities, and BFI could improve in these areas to make its large pool of information more accessible and thus more attractive• Sites such as Space

Collective and Surfrider Foundation do not offer much in the way of site utilities to engage users, and BFI could work to improve in this area and increase user involvement• BFI hosts such a wealth of

historical and current information regarding Fuller's Ideas, Life, and Designs that its redesign could establish BFI as the authority on information regarding the man

• Other sites' well engineered page designs and navigation mean that, in spite of the presence of advertising, users are more likely to visit those sites and thus spend less time at BFI.org• Several sites offer the site

utilities users have come to expect and BFI risks losing potential users to sites offering easier ways to get involved or stay informed• Other sites do a better job

of stating or making clear their organization's and/or website's purpose. Space Collective's textual description may be somewhat vague, but the design of its home page clearly implies the nature of the site's contents• BFI risks losing potential

members to sites such as Space Collective who have already succeeded in generating active member engagement

MethodsFirst Thinking and Then Doing

GUERILLA USABILITYeffectiveness + efficiency + satisfaction

Guess(ability) vs. Learn(ability)

EFFECTIVENESSCan a task be completed.

EFFECTIVENESSmeasured in yes or no.

EFFICIENCYhow long to complete a task.

EFFICIENCYmeasured in time.

SATISFACTIONphysiological / social / psychological /

ideological

SATISFACTIONmeasured in affect.

Guessability vs. Learnability

Guerilla Testing4 steps

1. Decide what to test...A whole site?Part of a site?A process?

2. Decide how to test it...What are the tasks to complete?What are the probing questions?

What are the post-test questions?

3. Test it...Recruit users

Test Users

4. Analyze results and make recommendations

QuantitativeQualitativeSeverity

dare to be naive.

[email protected]