Embed Size (px)
Citation preview
• Reports a card sort study conducted todetermine whether users rather than expertscould distinguish visual {in-)consistency in Webelements and pages
• Finds that color, background, and font were themain visual cues that users paid attention to
Testing the VisualConsistency of Web SitesTHEA VAN DER GEEST AND NICOLE LOORBACH
INTRODUCTION
Large Web sites are often created by teams ofdevelopers and designers, coming from differentdepartments and sub-organizations, representingdiverse product lines. A clear example of such a
large and complex corporate Web site is the Microsoft.comsite. The start page (Figure 1) consists of a collection oflinks to subsidiary sites, which carry' the information on thecompany's product lines. Three clicks on various linksshow three different subsidiary sites, all of them recogniz-able as belonging to the Microsoft Corporation, but alsoclearly distinctive from each other and from the start page.The consistent elements, such as the use of the color blue,the left hand navigation bar and the use of the logo createcoherence within and between pages, and between themain site and the subsidiary sites.
The Microsoft Corporation has developed detailedstyle guides and design templates for its various subsidiarysites and guards their application, whereas at the same timea certain level of variation is intended and promoted. Theumbrella organization uses this method to stress both thedistinction and the affinity between the various productlines or sub-organizations (van der Geest 2000). One maywonder whether users of these pages notice the mix ofconsistency and variation. Oo visitors of the subsidiary sitesstill recognize them as a site from the Corporation's family?Or are the visitors just confused, experiencing the variationas inconsistency?
VISUAL APPE,\R.4NCE AND CONSISTENCYVisual appearance has a major effect on how users appre-ciate Web sites. In a large scale study of Web site credibil-ity, Fogg, Soohoo and Danielson (2002) asked 2,684 peo-ple to rate sites and to comment on the aspects thatinfluenced the sites' credibility. The aspect mentioned mostoften was the sites' visual appearance. Nearly half of allparticipants in the study (.46.1%) referred spontaneously to
visual cues, such as layout, typography, and color schemes.Regrettably, comments on inconsistent visual design werenot scored separately by the researchers. But we can safelyassume that inconsistent visual appearance will lead tolower credibility. Interestingly, experts scored a sub-setof the same Web sites for credibility in a parallel study(Stanford, Tauber, Fogg, and Marable 2002). The expertswere far less concemed about the visual appearance ofthesites than the users. They rated the sites primarily on thebasis of the quality of the infonnation provided.
In another study, Ivory and Hearst (2002) investigatedthe relation between quantitative, automated measures ofWeb pages and quality ratings of experts. They found thatvisual aspects, such as the number of colors used, font size,and the presence of navigation bars, correlated with thequality rating of a site. Apparently, the looks of a Web pageare among the prominent, noticeable elements that directlyinfluence inteipretation and e\'aluation (Fogg and col-leagues 2002).
Handbooks and guidelines for interface designersstress the importance of consistency, including consistencyin visual appearance, mainly from the perspective of cre-ating usable interfaces and Web sites. In hi-s seminal bookon user interface design, Ben Shneidermann s first GoldenRule for interface design was "Strive for consistency" (1998,p. 74). In 1989, usability expert Nielsen edited a completebook on the issue, called Coordinating user interfaces forconsistency. Most o( the early investigators of interfaceconsistency focused on its beneficial effects on ease ofleaming for initial use of interfaces, and the transfer ofleamed behavior from one interface to the next.
Grudin (1989) challenged the maxim "Strive for con-sistency" and made a useful distinction among three types
Manuscript received 9 July 2004; revised 13 Octolxjr 2004;accepted 13 October 2004.
Volume S2, Number 1, Febnian' 200S • T«hiiirtlCO**flltlCATlON 2 7
APPLIED RESEARCHTesting the Visuai Consistency of Web Sites van der Geest and Loorbach
Figur« 1. Main site, subsidiary sites, and their consistency.
of consistency:• Intemal consistency of an interface design, that is,
consistency in physical and graphical layout, com-mand naming, dialog fonns, and so forth within aparticular product
• External con.sistency of interface features with fea-tures of other interfaces familiar to users
• Correspondence of interface feaaires to familiar fea-tures of tlie world beyond computing.In this articie, we will focus on the intemal consistency
of Web sites. In the world of Web page and site design, thegeneral interface guidelines about consistency are fre-quently repeated. For example, a set of research-basedWeb design and usability guidelines recommends "'present-lingj infonnation and similar functions consistentlythroughout the site, including logos, page titles, headers,navigation elements, etc. Also use a consistent position onall pages for logos, recurring text, buttons, and graphics"(National Cancer Institute 2004). The recommendations inthe handbooks and guidelines are suppoited by studies, forexample those by Ozok and Salvendy (2000; 2004), who
found that increasing the consistency of IinguLstic andphysical attributes of infomiational Web page interfacesresulted in fewer errors and improved performance byusers. So consistency seems to be desirable, but how toassess whether your site is consistent enough for yourusers, and whether deliberate variation is recognized assuch by site visitors?
ASSESSING CONSISTENCYThe literature about assessing consistency shows two ap-proaches: expert-focused and user-focused. In most casesreported, the interfaces or Web pages are evaluated byexperts, such as the designers themselves or usability ex-perts. The first part of this section describes in more detailhow well experts can assess consistency. Recently, a sec-ond approach toward asses.sing consistency has been pro-moted. Ozok and Salvendy (2000; 2001) developed a ques-tionnaire, the Interface Consistency Testing Questionnaire(ICTQ), to be filled out by u.sers. The details about testingconsistency with this questionnaire are presented in iliesecond part of this section.
2 8 TcdiJilc^iiCONMlNCATIOH • volume 52. Number 1, Febnmy S005
APPUED RESEARCHvan der Geest and Loorbach Testing the Visuai Consistency of Web Sites
r
Consistency evaluation by expertsHow well can experts assess and identify inccjnsistenciesthat might create problems for users of a Web site? There isample evidence that experts are not very successful inpredicting the users' seif-reported or observed problemswith texts, interfaces, or Web sites (Dieli 1986; Doubledayand colleagues 1997; Gahrmann 2004; Law and Hvannberg2002; Lentz and de Jong 1997). Also, among themselves,experts don t agree about the problems identified (Kessnerand colleagues 2001; Lentz and de Jong 1997). When de-tecting inconsistencies is explicitly reported, the expertsdetect many more inconsistencies than the users O ff esand colleagues 1991; Lentz and de Jong 1997).
Does that mean that experts report "false alarms," orcould it be that users overlook problems? Maybe inconsis-tency is one of those product features that experts canpinpoint but that users cannot or will not identify, espe-cially when the inconsistency does not lead to noticeableproblems with the execution of the experimental task. Wejoin in with the chorus of authors that advise to considerevaluation by experts and by users as complementaryrather than alternative ways of finding problems, particu-lady because expert review can help to find more skill- andaile-based problems, and testing with users can revealmore problems at the knowledge-based level of behavior(Fu and colleagues 2002; Law and Hvannberg 2002). Thatstill leaves open the question how to do an appropriateuser-centered evaluation of visual consisEency.
Consistency evaluation by users: The InterfaceConsistency Testing QuestionnaireOzok and Siilvundy (2000; 2001) developed and validateda questionnaire that measures the consistency of a Web siteby users. They distinguish nine factors that affect consis-tency, such as text structure, meaning, communicationalattributes, and physical attributes. The questionnaire con-sists of 94 yes-no questions which are scored on a seven-point scale, somewhat curiously labeled from "stronglydisagree" to "strongly agree." A few examples of questionsfrom the questionnaire are:
3- Are Ihe verbs used consistently?
64. Does tbe text confuse tbe user?
89. Are tbe sizes of tbe buttons, radio buttons andcombo boxes consistent?
Ozok and Salvendy claim that the questionnaire can beused to evaluate the consistency level of interfaces. Wehave our doubts about the usability of extended question-naires as Web evaluation instniments in general (de Jongand van der Geest 2000), but even more so about the
validity of this particular questionnaire. First of all. thewording of the instrument implie.s that inconsistency isalways bad, We claim that deliberate variation in verbaland visual elements of Web sites can, for example, be usedto indicate that the user is entering a new content area in asite. The instrument does not allow for this kind of "inten-tional inconsistency." Secondly, the questionnaire focusesstrongly on the consistency of the verbal content, not sur-prisingly since Ozok and Salvendy took a questionnaireabout Interface Language Consistency as their startingpoint. Out of 94, only 10 questions are about the visualappearance of the text on screen. 6 are about buttons anddialog boxes, and only 1 question is about the visualconsistency between various pages of a site. Given thepredominance of visual perception over verbal, one won-ders why the questionnaire focuses so much on consis-tency created by the words and sentences used, and yetclaims to measure interface consistency in general. Ourfinal concern about the validity of this approach: Filling outthe questionnaire results in a "site grade" for consistency.Such a grade might serve to assess the consistency of aparticular site against others (a summative evaluation goal),but neither the grade nor the questionnaire reveals whereexactly in the site the inconsistency occurs, and thus willnot be of any help to improve consistency or intendedvariation of the site (a formative evaluation goal). We aremore interested in an instmment that shows where u.sersactually experience inconsistency. In the remainder of thisarticle we will report our exploration of one user-centeredmethod, card sorting, for assessing the consistency of visualelements in a set of related Web sites and exploring users'interpretation of the visual cues that create consistency,
THE CARD SORT METHODCard sorting is a categorization and classification exercise.In a card sort study, the respondents are given cards withitems, for example content elements of Web sites. Thenthey are asked tt) order the cards in groups and give therule by which such allocation is made, by labeling andexplaining the groups they have created (Coxon 1999).Card sorting is an easy-to-explain procedure, and it lets therespondents come up with their own classifications andexplanations of what "belongs together." The researchersor designers do not impose their classifications and cate-gorizations on the respondents.
In Web site design, card sorting is often applied as auser-centered method for designing the information struc-ture of a site (Faiks and Hyland 2000; Fuccella and Pizzo-lato 1999; Rosenfeld and Mon,'ille 2002; Tnxister 2004; vander Geest 2001). For that purpose, the respondents sortcards with content items for the future Web site, and labelthe groups created. Both the grouping and the labelinggive insight in the users" mental model of the infonnationor services provided, and thus can serve as a basis for
'(>il1m(; .^2. Number I. Ft-bman' • Tt^cbniriilCOMMUMCATION 2 9
APPLIED RESEARCHTesting the Visuai Consistency of Web Sites van der Geest and Loorbach
designing the site staicture and the navigation features. Inother domains, such as marketing communication and an-thropology, card sorting is used as an elicitation tool withwidely varying types of items, such as sounds, pictures andstatements. In our case, we explored tlie method for itsappropriateness to reveal users' conceptions of the consis-tency of visual Web site elements,
THE RESEARCH DESIGNThrough the card sort exercise, we first of all wanted toanswer the following question:
Q 1 . Do users recognize (in-)consistency in visualsite elements and pages?This is the question that practitioners want to see answeredwhen ihey use the card sort method as a user-centered testmethod. As researchers we had two additional questions:
Q2. Can the users indicate and explain the visualcues that make site elements "belong together"?
Q3. If so, which visual cues create consistency?In the process of answering those questions, we wili
explore the vaiue of a visuai card sort for further researchinto the effeas of Web site elements.For this study, we selected a group of six sites. Tlie mainsite belongs to a Dutch consortium of higher educationinstitutions; five subsidiary sites belong to schools withinthe consortium. All sites were created with an organiza-tional style guide and templates provided by the consor-tiuni. Although there were obvious commonalities, eachschool had clearly intended to make its own site with itsown flavor and identity.
From each of the six sites, we selected visual pageelements like buttons, navigation bars, logos, text layout,and illustrations, for a total of 44 elements. We also selected24 typical pages, taken from the six sites. We selectedelements and pages because they either followed the styleguide or clearly deviated from it.
To ensure that the consistency of elements was notassesised on the basis of verbal rather than visual cues, wereplaced every word with a similar dummy word andreplaced pictures with a generic dummy picture. Figure 2ashows an original page; Figure 2b shows the adapted page.After the adaptations, tliere was no way to find out from tlietext, pictures, or pages that a particular element came froma specific school, such as, in this case, the School of Dance.
Twenty studenLs were selected as respondents in ourstudy. They all studied at the University of Twente, but hadno graphical design training or experience. Due to thestudent population in our university, we had mainly maleparticipants (]5 out of 20). Twenty respondents appears tobe appropriate for a card sort study CTullis and Wood 2004;Nielsen 2004). Although we were aware of the limitationsof using students as participants in our study, we think theyfomied a good enough sample of experienced Web users
Figure 2a. Page from subsidiary site.
BFigure 2b. Adapted page, in which ail verbal cues and thepicture are replaced by dummies.
for the exploratory study we were conducting.In the card sorting sessions, we first asked the respon-
dents to sort the 44 elements, which were printed on cards.They were asked to create groups of cards that they felt"belonged together." During and after the ordering of thecards, they labeled the groups with self-chosen labels. Wethen asked the respondents to group the 24 pages and labelthose groups. We also asked them to relate their groups ofelements with their groups of pages and label the relation-ships. This is the part of the study that practitioners wouldperform when doing a user-centered test of visual consis-tency.
But to explore the value of the meth(xl, we wentfurther. We interviewed the participants about each of iheirlabels, and we asked them to explain what made themdecide that items belonged together. Finally, we asked therespondents to take a Web-based test for color blindness,as we wanted to exclude color-blind respondents. None ofthe respondents proved to be color-blind.
3 0 1V(liiiir:tlC0MMUMCAT10N • Vohme bt Number I. Februarv 3005
APPUED RESEARCHvan der Geest and Loorbacb Testing the Visual Consistency of Web Sites
The card sorting session lasted 1 to I.S hours for eachrespondent. All respondents received a token gift for theirparticipation. '-'"'
Both the ordering ofthe elements and the grouping ofthe pages by the 20 respondents were analyzed with clusteranalysis (average linkage), a technique to detect pattems insets of orderings. We used EZSort (IBM, no date) to createthe typical tree diagrams that result from cluster analyses.That is what practitioners would do when they conduct avisual card sort. For the sake of this study, we also con-ducted a quantitative and qualitative analysis of the labelsand the interviews to find out how the respondents de-scribed and explained their groups.
USERS' RECOGNITION OF INCONSISTENCIES ' "First of all we wondered whether the respondents wouldrecognize consistencies and inconsistencies in the ele-ments and pages taken from the six related sites. Therespondents reported that they did not find it difficult togroup the cards with elements and pages, but some of therespondents had trouble labeling tlie groups and explain-ing their grouping principles. Our informal observationshowed that the respondents widely varied in the amountof time and attention they spent inspecting the cards.
If the respondents had recognized every element andpage as belonging together with the other elements fromthe same original site, they would have created six groupsof elements. This proved not to be the case. The 44 ele-ments were on average divided into 13 groups, with aminimum of 6 groups and a maximum of 23 groups. So our20 respondents had clearly very different opinions aboutwhat belonged together. The division of pages remainedcloser to the original six sites, with an average of 7 groups(minimum of 2 group.s, maximum of 9 groups).
The cluster analysis showed which elements and pageswere often placed in the same group, and thus belongedtogetlier in the perception of our respondents. Figure 3shows the cluster analysis results for the elements. At theleft hand, a list ofthe elements is shown. The abbreviationat the beginning of the element name shows its origin:Elements with the same abbreviation come from the sameoriginal subsidiary site.
The tree diagram shows the disagreemejtt among therespondents about whether elements belong together, If allor most of the respondents placed two elements in thesame group, those elements are connected at the far leftside in the tree, close to 0% disagreement. When very fewor none of the respondents placed two elements in thesame pile, the elements are connected at the far right side,clo.se to 100% disagreement. For example, the elementsthat came from the DA subsidiary site (in the middle of thetree) were grouped together by almost all respondents.These elements were internally consistent. But almost ev-
ElBinenten
« SW lihiM gm«n
W a . finedialut drti Mua^i
Figure 3. Cluster analysis results for the visual elements of
six related sites. • .
ery respondent had trouble sorting element 55EL (justabove the DA group) which almost never ended up in thesame group as any other element. This is a clearly incon-sistent item. Also, the other items ofthe EL site ended up inmany different groups, which shows that the respondentsdid not recognize the elements of the EL subsidiary site as
Volume 52, Nuinber I, Febnnuy 3005 • T e c h m c a l C O M M M C A T I O N 3 1
APPUED RESEARCHTesting the Visuai Consistency of Web Sites van der Geest and Loorbach
T4BLE f: LABEUNC STRATEGIES OF USERS
Figure 4. Cluster analysis results for the pages of sixrelated sites.
consistent.At the page level, the respondents agreed to a great
extent about what belonged together. Figure 4 shows theresulting tree diagram for the pages.
For a few pages, such as 48DA and 64PED, it was notcompletely clear to all respondents where they belong.Tho.se pages are candidates for further consistency inspec-tion.
From the cluster analysis ofthe elements (Figure 3) wecan conclude that 9 out of the 44 elements are particularlyinconsistent, with 70% or more disagreement among therespondents. Four out of those nine elements came fromone subsidiary site (EL), which is particularly inconsi.stent.Two subsidiary sites (EL and PED) are confusingly similar.From the page level analysis (Figure 4), we can concludethat the six sites are distinctive, but two pages, a PED pageand panicularly a DA page, are somewhat confusing.
We started this analysis to find out whether a visualcard sort is an appropriate technique to reveal the users'perceptions of consistency within and among sites. Wethink it is. A visual card sort with subsequent cluster anal-
Descriptive '
Associative
Sender/function
Unlike others
Combination
Other - -
Total
Elements
N
133
42
34
11
40
4-
264
%
50
16
13
4
15
2
100
Pages
N
79
10
25
1
20
0
135
%
59
7
19
1
15
0
100
ysis is a useful alternative or complement to consistencyevaluation by experts. The procedure is not very compli-cated Of time-consuming, and can be executed by practi-tioners who are not trained researchers.
USERS' LABELS AND EXPLANATIONS OF CONSISTENCVWe also wondered how our users would label and explaintheir groups—in other words, which cues they perceivedas creating the sease of "belonging together." During thecard sort exercise, we provided sticky notes for respon-dents to write their labels. Some respondents chose to labelgroups with one-word titles, whereas others came up withlong and detailed distinctions. We analyzed the labels theyattributed to their groups and used the interviews as back-ground information for interpreting tlie labels.
We saw five different types of labels applied.1 • Descriptive labels (orange, horizontal lines, and
so forth)2. Associative labels (.scientific, tranquil, and so
forth)3 . Sender- or function-oriented labels {sports club,
product information, and so forth)4. Unlike-the-others labels (for example, a style of
its oum)5. Combination labels (for example, color, aggres-
sion).All together, our 20 respondents created 264 groups ofelements, and 135 groups of pages. Table 1 shows howoften the four types of labels were applied to name thegroups.
In the next two parts of this section, we will focus onthe two dominant labeling strategies, descriptive and asso-
3 2 Terlinii.dCOMMltiCATION • Voliimf W.Numhrr I,Fcbniary2M5
APPUED RESEARCHvan der Geest and Loorbacb Testing the Visual Consistency of Web Sites
TABLE 2: ViSUAL CUES iNDESCRIPTIVE LABELS OF ELEMENTS
TABLE 3: VISUAL CUES INDESCRIPTIVE LABELS OF PAGES
Color
Background
Font
Illustration
Grid/Navigation
Logo
Number ofcues
(N = 288)
139
42
41
31
. m6
Mentioned asfirst cue(N = 189)
m,. 14
.. ^" ^. \2
Color
Background
Font
niustration
Grid/Navigation
Logo
Nunnber ofcues
(N = 210)
92
,: , 19 '
19
5
74
'I'
Mentioned asfirst cue(N = 113)
6
1
1
31
0 ,
ciative labels. We will also consider the labels assigned tothe pages and the question whether differences in labelsreflect a true difference in groups.
Descriptive labels of elementsThe most common way lo indicate that elements belongtogether is by describing their physical characteristics. Ta-ble 2 shows the visual cues that were mentioned in thedescriptive labels. Often one label referred to various de-scriptive cues, for example font color. In Table 2, suchcombined cues have been counted twice, once under Fontand once under Color. This explains why a total of 133descriptive labels provided 288 visual cues. The Grid/Nav-igation category contains all references to the physicalappearance of buttons, links, menus, navigation bars, lines,frames, text layout, and screen layout.
According to our respondents, color is the dominantcue to group site elements as "belonging together." It wasnot only mentioned most often; it was also very oftenmentioned as the first part of a label.
Associative labels of elementsElements were sometimes categorized on the basis of somekind of association, such as warm or chaotic. This labelingstrategy was mainly applied by 6 out of the 20 respondents.These six used this strategy quite often, creating 36 out ofthe 42 associative element labels. The other respondentshardly ever referred to an association to label a group, Wealso saw quite a few combination labels (29 out of 40) inwhich associative cues appeared with other cues. Respon-dents, for example, combined descriptive cues like orangewith associations like warm in one label,
We find the associative labeling strategy interesting fordesigners of organizational brands. The design process ofan organizational image often starts with defining the as-sociations the organization wants to evoke in its commu-nication: dynamic, reliable, caring, and so forth. Cues aboutsuch associations appear to be perceived by just a part (6out of 20) of our respondents. And these respondents alsodisagree about the nature of the association. For example,a particular element that ended up a few times in groupswith associative labels was one time placed in a groupcalled strange, by another respondent in a group calledartistic, and by yet another respondent in a group calledclear. Although defining corporate identity might be theusual start of the visual design process, it appears that itdoes not give direaion in deciding on the actual physicalcharaaeristics of the interface.
Labels of pagesThe respondents in our study also created groups of pages,labeled those groups and explained their labels. Just aswith the elements, most labels were descriptive (79 out of135, or 59%). Another labeling strategy was to identifypages as belonging together because they seemed to havethe same sender or function (25 out of 135, or 19"/(i). Fourout of 20 respondents were responsible for almo.st all thesender-/function-oriented labels.
Tlie descriptive page labels are shown in more detail inTable 5- The cues are assigned as explained above.
Just as at the element level, the dominant descriptivecharacteristic to identify pages as "belonging together" iscolor (92). It was mentioned most often, and when it wasmentioned, it appeared often as first cue. Also, character-
Vdume 52. Number 1. Fcbiuaiy 2005 • TethnitalCOMANCATION 3 3
APPUED RESEARCHTesting the Visuai Consistency of Web Sites van der Geest anci Loorbacb
Lstics of the pagt- grid and navigation devices were fre-quently mentioned (74 times), much more often than atelement level, as one might expect.
It surprised us that the logo, which was visible onmany of the pages, was used only once to indicate thaipages "belonged together." The logo was a combination ofa graphic and type, which in some pages was placed at theleft, in other cases at the riglit hand side. In all cases butone the logo was wliite, and could be easily distinguishedfrom tlie various background colors used in the sites. Yetour respondents did not identify it as a feature that createdconsist enc7.
Different labels, different groups?At tile element level, b out of 20 resp<jndents created thelarger part of all associative labels. At the page level. 4out of 20 respondents were responsible for almost allsender-/function-oriented labels. We checked whether theserespondents deviated in their grouping of ihe cards fromthe respondents who applied a different labeling strat-egy. A comparison of the tree diagrams of those respon-dents that applied an associative labeling strategy for theelements with ihose that did not showed that the samegroups were discerned, independent of labeling strat-egy. The same proved to he true when we compared thetree diagrams of the four respondents who labeled thepage groups on the basis of function/sender with thetree diagrams of the other respondents. Again, the samegroups were discerned, even when they were labeleddifferently.
So even when respondents classify elements or pageson a different criterion (for example, associative warm.instead of descriptive orange), they still discern the samegroups of elements as belonging together. We tliink thatthis finding is i-eassuring for designers who want to w(5rk Iroman associative or tiirget grtjup perspective to create a brand ororganization image. It is possible to create site dements andpages from such a perspective and be consistent.
RECOMMENDATIONS FOR DESIGNERSOn the basis of this study we can make some recommen-dations to those of us who are responsible for guarding orassessing the cfsnsistency within or between Web sites.
1. Be consistent.Althougli we did not investigate the efTect of inconsistencyon perfonnance or error rate, there is enough evidence toshow that inconsistency can be detrimental. This studysh(5ws that inconsistency can be assessed by users, eventhough tliey do not mention inconsistencies as problems inother types of studies, such as usability studies in which theyverbalize their thoughts and actions while perfonning t:isks.
2. Use color to signal within-siteor between-site coherence.According to the self-reports of our respondents, color isthe most important signal to indicate that Web page ele-ments or pages belong together. This included not only thecolor ofthe background and bars in the screen design, butalso the color of buttons, fonts and lines. A deliberatevariation in colors or hues can go a long way to signal usersthat they are entering a new content area or a subsidiarysite. However, overuse of different colors should beavoided (Ivory and Hearst 2002). Also, it should be notedthat preferences for particuiar colors might be tied to cul-ture. Finally, when applying color, designers sh(juld be awareof the limitations of color use for people who are colorblind(alxjut 1 out of 12 male site visitors!). The Vischeck Web sitewill display your site as it is perceived by people with threedifterent color deficiencies (Vischeck 2003).
3. Use grid and navigation elementsto create consistency at page levei.At the page level, designers should signal consistencythrough the visual elements that show in the Gestalt of thepage, such as the placement of blocks of color and texts;similarity in shape, size, and colors; and application ofspacing and lines. The users' first glance at the page shouldalready reveal its consistency or intended inconsistency. Tosimulate the users' first glance, designers should use the"squint test": When various Web pages are viewed withsquinted eyes, the pages should be mutually consistent. Asimilar test can be conducted by "blurring" the pages inimage editing software such as Photoshop.
4. Don't reiy soiely on logos to createa persistent organizational identity.In our study, the logo proved to be only a minor cue forconsistency lietween pages. Very few respondents referredto die logo as a cue to group pages as 'belonging together."This result might be due to the presence of more powerfulcues, such as color anci page grid. It might also be causedby the qualities ofthe particular logo used on the pages inour study. Whatever the reason for the limited effect ofrepeating the logo, our study shows that a logo does notcreate consistency by itself. Designers should supplementlogos with other visual cues to constaict a consistent orga-nizational identity throughout Web pages.
I
5. For user-centered formative evaluationof consistencies between and within Websites, consider a visual card sort.As has been demonstrated, the visual card sort offers a clearview of what elements and pages are inconsistent or con-fusingly similar to each other. When the goal is just detect-ing inconsistencies, practitioners can limit themselves to
3 4 Jr, liriir,ilC0KMlMCA110N • Volume 53. Mumber I. Februaiv 2(m
APPUED RESEARCHvan der Geest and Loorbacb Testing the Visuai Consistency of Web Sites
asking the respondents to sort and label the cards. Espe-cially when the tester disposes of the rougli visual designeiements files, replacing the verbal cues is not too cumber-some. The time-consuming individual interviews and ob-servations that we conducted are not necessary for a simpleuser-centered consistency check. And the visual card soncan be conducted even more efficiently in group sessionsor at a distance, using programs such as WebSort (Wood,Wood and Anderson 2002) or EZSort (IBM. no date).
RECOMMENDATIONS FOR RiRTHER RESEMCHThe visual sort method that reveals u.sers' ideas about thevisual elements in Web sites that belong together offersgood perspectives for further research. Stripping away allthe overt language and graphic elements to render every-thing that rt^mains as the visual appearance can be a spring-board for further studies into the nature and impact ofvisual Web site elements.
In tliis study, we fcxrused on the questitin whether usersrecognized visual consistency. Furtlier studies could relate thevi.sual elements and their inconsistencies on the one handwitii effects such as for example navigation behavior, credi-bility, or global quality ratings on the other hand. The visualcard sort would help the researcher to single out the particularvisual elements tliat could be niiinipulated (made consistentor kept inconsistent) in various experimental versi(jns of asite. Such studies would deepen our iasight into the effects ofthe various visual elements that now remain hidden under tlieumbrella laliel ''visual appearance."
Anotlier issue that could lie investigated with the visuaicard sort niethcxl is the relationship between visual prefer-ences and cultural background, In this study, we focusedon the visual elements that a narrow group of participants,Dutch university students, found consistent. A comparisonof their visual sorts with those of people from differentbackgrounds or nationalities could reveal whether thereare any cross-cultural differences in visual consistency per-ception and appreciation. Marcus and Gould (2000) claimihat people from cultures with a liigh tendency to avoid risksand uncertainties prefer Web sites that are very coasistent. Across-cultural compari.son of visual card sort results could helpus to investigate the value of such claims, Knowing whethertliere are universal pattems of interpretation of visual ele-ments and visual consi.stency Is very useful for designers ofWeb sites for intemational audiences.
Olwiously, the visual card sort metlicxl is an easy way toperfomi user-centered testing of tlie visual coaslstency (jf Wehsites. But tlie method also offers perspectives for exploring theeffective characteristics of visual appearances of Web sites. TC
ACKNOWLEDGMENTSWe thank the Fontys Consortium of Schools in the Netherlandsfor letting us use their site and subsidiary sites for our study.
We also thank two anonymous reviewers for their thoughtfuiand helpful comments.
REFERENCESCoxon, A. P. M. 1999. Sorting data: Collection and analysis.
Thousands Oaks, CA: Sage.
de Jong, M., and T. van der Geest. 2000. CharacterizingWeb heuristios. Technical communication 40:311-326.
Dieli, M. 1986. Designing successful documents: Aninvestigation of document evaluation methods. PhDdissertation, Carnegie Mellon University.
Doubleday, A., M. Ryan, M. Springett, and A. Sutcliffe. 1997.A comparison of usability techniques for evaluating design.Proceedings of the ACM/DIS conference. Amsterdam,Netherlands: Association for Computing Machinery, pp.101-110.
Faiks, A., and N. Hyiand. 2000. Gaining user insight: A casestudy illustrating the card sort technique. College andsearch libraries 61:349-357.
Fogg, B. J., C. Soohoo. and D. Danielson. 2002. How dopeople evaluate a Web site's credibility? Resuits from a largestudy. New York, NY: Consumer Webwatch. http://www.consumerWebwatch.org/news/report3_credibilityresearch/stanfordPTL.pdf.
Fu, L., G. Salvendy, and L. Turley. 2002. Effectiveness ofuser testing and heuristic evaluation as a function ofperformance classification. Behaviour and informationtechnology 2^:^37-^43.
Fuccella, J,. and J. Pizzolato. 1999. Giving people what theywant: How to involve users in site design. Formerlyavailable online from IBM.
Gahrmann, S. 2004, De test getest [The test tested]. Master'sthesis. University of Twente.
Grudin, J. 1989. The case against user interface consistency.Communications of the ACl\^ 32:1164-1173.
IBM. n.d. EZSort (beta release), http://www-3.ibm.oom/ibm/easy/eou_ext.nsf/publish/410.
Ivory, M. Y.. and M. A. Hearst. 2002. Statistical profiles ofhighly rated Web sites. Paper presented at CHI 2002conference, Minneapolis MN.
Jeffries, R., J. R. Miller, C. Wharton, and K, M. Uyeda.1991. User interface evaluation in the real world: A
Volume b2. Ntimbcr 1. Febiuar>' 3005 • TechnicalCOMMlMCATION 3 5
APPUEO RESEARCHTesting the Visual Consistency of Web Sites van der Geest and Loorbach
comparison of four techniques. Proceedings of the ACMCHI'91 conference. New York, NY: ACM Press, pp. 119-124.
Kessner. M., J. Wood. R. F. Dillon, and R. L West. 2001. Onthe reliability of usability testing. Interactive posterpresented at the CHI 2001 conference, Seattle. WA.
Law, L.-C, and E. T. Hvannberg. 2002. Conriplementarity andconvergence of heuristic evaluation and usability test: Acase study of UNIVERSAL brokerage platform. Paperpresented at the NordiCHI conference, Aarhus, Denmark.
Lentz, L., and M. de Jong. 1997. The evaluation of textquality: Expert-focused and reader-focused methodscompared. IEEE transactions on professionalcommunication 40:224-234.
Marcus, A., and E. W. Gould. 2000. Crosscurrents: Culturaldimensions and global Web user-interface design.Interactions 7(4): 32-46.
National Cancer Institute 2004. Research-based Web-designand usability guidelines, http://usabiiity.gov/.
Nielsen. J., ed. 1989. Coordinating user interfaces forconsistency. Boston, MA: Academic Press; rpt. SanFrancisco, CA: Morgan Kaufmann Publishers, 2002.
. 2004. Card sorting: How many users to test.Alertbox. 19 July, http://www.useit.com/alertbox/20040719,html.
Ozok, A. A., and G. Salvendy. 2000. Measuring consistencyof Web page design and its effects on performance andsatisfaction. Ergonomics 43:443-460.
. 2001. How consistent is your Web design?Behaviour and information technology 20:433-447.
. 2004. Twenty guidelines for the design of Web-based interfaces with consistent language. Computers inhuman behavior 20:149-161.
Rosenfeld, L.. and P. Morville. 2002. Information architecturefor the World Wide Web. 2nd ed. Beijing, China: O'Reilly.
Shneiderman, B. 1998. Designing the user interface. 3rd ed.Reading, MA: Addison Wesley.
Stanford, J., E. R. Tauber, B. J. Fogg, and L. Marable.2002. Expert vs. online consumers: A comparativecredibility study of health and finance Web sites. NewYork, NY: Consumer Webwatch.http://www.oonsumerWebwatch.org/news/report3_credibilityresearoh/slioedbread.pdf.
Trooster, S. 2004. Practices and principles of designinginformation structures for target audiences. Master'sthesis. University of Twente.
Tullis, T., and L. Wood. 2004. How many users are enoughfor a card-sorting study? Paper presented at the UsabilityProfessionals Association conference, Minneapolis, MN.
van der Geest. T. 2000. Web design is communicationdesign. Amsterdam, Netherlands, and Philadelphia, PA:John Benjamins.
. 2001. De informatiestructuur van Websites. Desorteertaak als hulpmiddel voor de site-architect [Theinformation structure of Web sites. Sorting as an aid forthe site architect]. [Tekstjblad 2(1): 36-41.
Vischeck. 2003. http://www.visoheck.oom.
Wood, L,E,. J. R. Wood, and J. Anderson. 2002. Web-basedcard sorting for information architecture. Presentation atthe Northern Utah Human Computer Interaction SpecialInterest Group meeting, http://www.acm.org/chapters/nuchi/2002/09mtg.html.
THEA VAN DER GEEST is an associate professor in theCommunication Studies/Technical Communication Depart-ment of the University of Twente. the Netherhinds. Sheteaches courses in interface and interaction design. Web de-sign and research methodology. Her research focuses on theprocess of Web design, methods for evaluating Web sites,usability and accessibility, and information and documentdesijjn for new media. Contact information:[email protected].
NICOLE LOORBACH is a junior researcher tn the Tech-nical Communicati{)n Department at the University ofTwente. the Netherlands. Her re.search intere.sts include textand document design, in particular motivational elements intechnical documents. Contact information:[email protected].
3 6 U-ihrii<.,ilCOMUIlMCATX)N • Vdume 52. Number I.Fcbmarv 3005
