Usability processes in open source projects

SOFTWARE PROCESS IMPROVEMENT AND PRACTICESoftw. Process Improve. Pract. 2006; 11: 149–162Published online in Wiley InterScience (www.interscience.wiley.com). DOI: 10.1002/spip.256

Usability Processes in OpenSource Projects

Research Section

David M. Nichols1,*,† and Michael B. Twidale2

1 Department of Computer Science, University of Waikato, Hamilton,New Zealand2 Graduate School of Library and Information Science, University of Illinoisat Urbana-Champaign, IL, USA

We explore how some open source projects address issues of usability. We describe themechanisms, techniques and technology used by open source communities to design and refinethe interfaces to their programs. In particular we consider how these developers cope with theirdistributed community, lack of domain expertise, limited resources and separation from theirusers. We also discuss how bug reporting and discussion systems can be improved to bettersupport bug reporters and open source developers. Copyright 2006 John Wiley & Sons, Ltd.

KEY WORDS: open source; usability; interface design; bug reporting

1. INTRODUCTION

In contrast to the strong claims for the power,flexibility and robustness of open source software(OSS), its usability has at times been consideredto be relatively weak (Benson 2004; Raymond1999). Previous work has considered this issue andexplored reasons why it might be so, and how itcould be alleviated (Nichols et al. 2003; Nichols andTwidale 2003).

In this article we explore how two open sourceprojects (Mozilla and GNOME) explicitly try toaddress the challenge of integrating sensitivity tousability into the open source development process.An examination of some of the methods used in theprocess of improving their usability, strengths andweaknesses can, we believe, serve as a starting pointfor designing better processes and technologiesto support greater adoption of usability practicesthroughout the open source world.

∗ Correspondence to: David M. Nichols, Department of Com-puter Science, University of Waikato, Hamilton, New Zealand†E-mail: [email protected]

Copyright 2006 John Wiley & Sons, Ltd.

In Section 2, we give an outline of the previouswork on open source usability. In Section 3 weoutline the sources, methodological issues andexamples of current practice. Section 4 describesthe main findings and is followed by discussionand conclusion.

2. BACKGROUND

The growing prominence of OSS over recent yearshas led researchers to examine how open sourceprocesses differ from those of conventional softwareengineering. This includes the examination of manyaspects of open source development including moti-vation (Hars and Ou 2002), source code repositories(Mockus et al. 2002), sponsorship and governance(O’Mahony 2003) and requirements (Scacchi 2002).A smaller body of work has focussed on issuesof usability (e.g. Benson 2004; Benson et al. 2004;Nichols and Twidale 2003; Twidale and Nichols2005). This research has been motivated by a beliefthat usability is an area of weakness for OSS,although there are few direct comparisons withproprietary applications to make clear judgments(Nichols and Twidale 2003). The main arguments

Research Section D. M. Nichols and M. B. Twidale

for the proposition that usability may be an area ofconcern for OSS can be summarized as follows:

• Developers are not typical end users.• Usability experts do not get involved in OSS

projects.• Open source projects lack the resources to

undertake high-quality usability work.• Interface (re)design may not be amenable to the

same approaches as functionality (re)design.

One of the strengths of open source develop-ment, and the academic collaboration that precededit (Feller and Fitzgerald 2002), has been that thedevelopers and users of the software have been verysimilar groups. Raymond (1998) comments that‘Given a large enough beta-tester and co-developerbase, almost every problem will be characterizedquickly and the fix obvious to someone’. This ‘useras co-developer’ approach has proven very success-ful for creating development tools and server-sideapplications (Nichols and Twidale 2003; Feller andFitzgerald 2002, p. 114). This success is at leastpartly derived from the similar experiences of themembers of the developer community. When theuser group is distinct from the developers, usabilityproblems can arise; assumptions that apply withinone community do not apply to the other. The entirefield of user-centred design (e.g. Nielsen 1993) is aresponse to this gap between developers and users.As OSS moves to creating applications that ‘nor-mal’ people may want to use, the developer–usergap becomes apparent and the OSS developmentmethodology is questioned (Frishberg et al. 2002;Nichols and Twidale 2003; Raymond 1999). Thechallenge of producing quality web browsers, officesuites and graphics tools is qualitatively differentfrom writing a good web server.

Open source projects that follow the archetypeof Raymond’s (1998) developers ‘scratching anitch’ typically have not included members withsignificant usability expertise (Nichols and Twidale2003; Raymond 1999). Possible explanations includea shortage of usability engineers and differences inculture between the disciplines of programmingand usability (Nichols and Twidale 2003). Evenlarge open source projects have had problems inthis area: one of the ‘lessons learned’ in the Mozillaproject was to ‘ensure that UI [user interface]designers engage the Open Source community’(Trudelle 2002). The lack of human–computerinteraction (HCI) expertise in OSS projects is in

part due to the volunteer nature of the community;in contrast, companies employ usability engineers.In the absence of usability expertise, ‘itch-based OSSdesign’ will not necessarily succeed in improvingease of use for those users, such as novices, whosebackground differs from that of the developers(Nichols and Twidale 2003).

Many open source projects have developed with-out significant commercial input. While this showsthe possibility for quality software to be writtenoutside a commercial environment, many projectsare naturally at a significant resource disadvantagecompared to their proprietary rivals. However, asopen source has gained in prominence, companieshave started to engage with OSS projects and haveprovided significant resources to some projects. Forexample, Sun Microsystems has contributed to theGNOME project (German 2003) through laboratory-based usability tests (Smith et al. 2001) and thedevelopment of a set of human interface guidelines(HIG) (Benson et al. 2004).

One approach for commercial involvement ininterface development is simply to layer a pro-prietary interface over an open source base, e.g.Apple’s OS X. A related approach is to create acommercial partial ‘wrapper’ around one aspect ofOSS functionality that is widely perceived to becomplex, such as installing Linux. However, theseprojects are the exceptions, and most are seriouslyconstrained by lack of resources and usability exper-tise: ‘few (open source) projects yet meet the mostbasic usability requirement: a thorough understand-ing of their target audience’ (Benson 2004).

The overall picture, then, is of a community whichis only recently starting to engage with the conceptsof user experience design and usability engineeringwhich have been common elsewhere for over adecade (Nielsen 1993). The open source communitybrings characteristics that are unusual in the worldof HCI: extreme geographical distribution, lackof expertise, a code-centric world view, lack ofresources and a culture that can be somewhat aliento interaction designers. In the rest of this article weexamine how these two cultures currently intersect.

3. USABILITY IN OPEN SOURCEDEVELOPMENT

Although, as the literature reviewed above notes,many OSS projects fail to take account of usability

Copyright 2006 John Wiley & Sons, Ltd. Softw. Process Improve. Pract., 2006; 11: 149–162

150

Research Section Usability Processes in Open Source Projects

issues or do so in a rather cursory manner, there areprojects that take usability seriously, with a concernfor designing systems that will also be usableby less technical people than those volunteeringto participate in the development process. Howdo those projects manage despite the barriers tousability outlined above? What can be learned fromwhat is currently done, in contrast to criticizingwhat is currently not done or done rarely? Howcan current usability processes be improved andpropagated to other projects? Over the past threeyears we have done an informal study of the Mozillaand GNOME projects, which do explicitly addressusability. Unlike much research on open sourcethat uses quantitative measures (e.g. Herbsleb andMockus 2003; Mockus et al. 2002), we followed anobservational approach (e.g. Sandusky et al. 2004;Scacchi 2002) based around publicly available bugrepositories, mailing lists and blogs.

We wanted to know more about the kinds ofusability processes that these projects undertake.These processes include activities as basic as dis-cussions about usability analysis and design. Webelieved that a good starting point was to try andunderstand what was talked about and how. Welooked for recurrent themes as well as gaps in thediscussions and aspects that appeared particularlyproblematic or clumsy and therefore amenable todesign interventions either in the form of bettersupport technologies, or different processes, per-haps involving other kinds of participants. We alsolooked for noteworthy similarities and differencesfrom two related processes that are better stud-ied and understood: OSS functionality discussionsin the same projects and the idealizations of howusability engineering methods as described in theHCI literature.

This study is preliminary. Our aim was not tocreate a definitive overview of the state of usabilitypractices across the whole of OSS development. Itwas not even to give a complete picture of practicesin two chosen projects. Rather, it was to gain aninitial understanding of some of the issues involved,in sufficient depth to be able to comprehend whatwas done and why, and what might be designed toimprove things in the future. We can make no claimthat the issues uncovered are universal or the mostimportant. But we do aim to use them as hypothesesfor future, more systematic work.

Our exploration of this incredibly rich datasource was entirely opportunistic, rather than an

attempt at a systematic investigation, exhaustiveaggregation or careful sampling. Such methods willbecome more appropriate now that we have atleast an understanding of the nature of some ofthe issues that merit more detailed investigation.Furthermore, resource constraints and our ownmethodological research interests led us to explorerapid lightweight methods that trade coverageand rigor for speed and flexibility, while allowingsubstantial detail of particular design processes tobe investigated.

This openness of OSS development allowsresearchers to see much of the software develop-ment process that is normally kept hidden withinthe walls of commercial buildings. It is unfair tocompare imperfect but public OSS processes withimagined but concealed commercial processes. It ishardly the case that usability is a resolved prob-lem in closed source software projects. There aremany stories of commercial projects that failed totake usability engineering seriously or allocate suf-ficient resources to it, yielding dreadful interfacesto unusable systems. Evidence from other projectsover many years shows that real life constraintsfrequently enforce departures from the idealiza-tions of software development textbooks (Austin2001; Gilb 1989). This may happen in closed sourcedevelopment just as much as in open source.

Below we discuss three issues in current OSSusability that emerged from our data analysis:reporting a problem, complexity management andthe use of explicit interface guidelines.

3.1. Reporting a Usability Problem

Even the basic act of reporting a usability bug ismore complex than reporting a functionality bug. Inthe case of the former, almost by definition the useris confused about what happened or what to do next(unless reporting a potential problem that a thirdparty might have). Articulating that confusion canunderstandably be difficult. Additionally, currentbug reporting software, such as Bugzilla, seemsto be inadequate for supporting effective usabilitydiscussions for various reasons:

3.1.1. Difficulties that a User May Experience with aGraphical User Interface May Not be Easy to DescribeTextually:A purely textual medium is of course not ideal fordiscussing problems with the use of a graphical


151


UI. Nevertheless people cope by workaroundsthat deal with aspects of the problem at thecost of extra work. This rather detracts fromthe need for lightweight participation from abroad spectrum of the volunteer community. Itis possible to include screenshots within Bugzillareports, but it is an extra burden both to createthem and to read them (distracting from themain flow of the textual comments). Furthermore,even a screenshot and accompanying text maybe insufficient to disambiguate the subject of acomment (see Figure 1). Contrast this with thecommunication of the same problem face-to-face,which could be done in one second with a pointingfinger and a grimace.

One workaround to this problem is the use ofannotations on screenshots to indicate the locationof the perceived problem. Unfortunately the useof screenshots also can raise issues of privacy,

Figure 1. Clarification dialogue over a screenshot fromthe GNOME Bugzilla (http://bugzilla.gnome.org)

requiring yet more work to resolve. Twidale andNichols (2005) report on the use of blurringto conceal personal information in a screenshotattached to a bug report. Many bug reports donot have this level of detail and so clarifyingdialogues are frequently necessary. Given all thiseffort, it is hardly surprising that only a few peoplebother to make error reports particularly of usabilityproblems.

3.1.2. Some Usability Confusions Have a DynamicElement:The confusion can be an aspect of an action sequencerather than being reducible to a talk about singlemenu or dialogue box. In such cases even a singlescreenshot may be insufficient, no matter how easyit is to create or use.

3.1.3. Proposed Design Solutions are Often BestExplained and Justified Using Non-Textual Means:This can involve creating mock-ups in HTML, aninterface design toolkit such as Glade, a standarddrawing package, simple sketches, or even, andto us most surprising, the use of ASCII art(Twidale and Nichols 2005). The presence ofeffortful ASCII art is a good indicator that even intextual environments the nature of interface designrequires graphical communication.

3.2. Complexity

Usability bug reports and their subsequent analysesand re-design discussions have the potential tobe particularly complex and contentious (Twidaleand Nichols 2005). We believe that this additionalcomplexity and contentiousness, compared to manyfunctionality bugs, may occur for a number ofreasons:

• The very existence of certain usability bugs may beviewed as subjective and hence unreliable: Usabilitybugs may not be afforded the same statusas functionality bugs, particularly functionalitybugs that cause crashes. A given interfaceelement can be confusing or ambiguous tosome people but not to others – something weterm ‘subjective’ usability bugs. These subjectivebugs are in danger of being tagged withthe Bugzilla status ‘WORKSFORME’, which isconventionally used to note that a functionalitybug reported by one person cannot be replicatedby another. In this case it is used to mean:


152


‘you may be confused but I’m not, so thereis no problem’. This is a useful approach foroverall complexity management (handling un-replicable bugs), but does create a bias againstusability bugs.

• Usability bugs can cluster in complex ways: Often ausability bug can have similar manifestations inother parts of the interface, and this is importantto note if within-application consistency is tobe maintained. Similarly a fix for one usabilityflaw can have implications elsewhere in theoverall interface design. As a result multiplebugs can be related, and meta-bugs are used tohelp manage the complexity of the bug network(Sandusky et al. 2004). However, these can leadto parallel discussions of the same topic inmultiple locations.

• Discussions need to involve complex design ratio-nales, multiple, perhaps contradictory goals anddesign trade-off s: Usability bug analysis and fixdiscussions can become long, and it can behard to keep track of all their elements. Else-where, (Twidale and Nichols 2005) we describean example from Mozilla’s preferences panels,where a proposed fix improves the consis-tency of one aspect of a dialog box, but hasimplications for other dialog boxes, as well asraising issues of general cross-platform flexi-bility. The discussion also touches on a widerOSS functionality-usability-consensus problem,namely, how alternate design solutions are fre-quently resolved by providing both as alternateoptions, and this in turn leading to ever morecomplex configuration options and interfaces.Alternate design solutions should be debated

in the light of these multiple goals and trade-offs, but later discussions may focus on onetrade-off issue and omit factors critical to theunderstanding of the rationale of prior designdecisions.

Our assessment of the relative complexity ofusability and functionality bugs, assuming a cleardistinction can be made, is still qualitative in nature,and obtaining quantitative evidence is still an issuefor further research. As we describe later, thatevidence in fact would need to be drawn fromsources beyond project bug repositories.

3.3. Human Interface Guidelines

HIGs are one way to guide usability work, bothdesign and discussions, about candidate designalternatives. HIGs are common in commercialsoftware development and Sun Microsystems havemade significant contributions to the GNOME HIG(Benson et al. 2004). Shows an example of a bugreport referring to the HIG. The HIGs can be aneffective means to pre-empt a long discussion, asthey act as a source of authority on what changesshould be made.

The use of the HIG in GNOME is a concreteexample of the ‘rational culture’ of Yamauchi et al.(2000) wherein members of open source projects‘try to make their behaviour logically plausible andtechnologically superior options are always chosenin decision-making.’ The HIG provides explicit(presumably best practice) guidance on issues ofinterface design enabling the community to orientitself towards ‘action’ (Yamauchi et al. 2000); thedialog should be re-designed this way (Figure 2).

Figure 2. A bug report using the human interface guidelines (HIG) from the GNOME Bugzilla (http://bugzilla.gnome.org)


153


The examples above illustrate the additionaldifficulties encountered in undertaking usabilitypractices in OSS development. In the next sectionwe attempt to systematize those difficulties and usethat as a framework for understanding the existingpractice and possible future improvements.

4. ENHANCING REPORTING, ANALYSISAND (RE)DESIGN

Given that there are difficulties with addressingusability, in this section we explore how reporting,analysis and design work occur in current opensource projects. By an analysis of the componentsof usability processes and their current manifesta-tions as practices, workarounds and appropriationsof available technologies, we aim to suggest mech-anisms to improve the current practice.

4.1. Improving Bug Reporting

Knowing that there is (or may be) a problem withthe system is of course a prerequisite for re-design.Raymond (1998) notes that ‘given enough eyeballs,all bugs are shallow’, but with usability bugs, it iscritical to have the right kind of eyeballs. That meansinvolving usability experts, but also end users whomay be willing to note their confusion or frustration.Participation in bug reporting is acknowledgedto be by a tiny percentage of all OSS users.Nakakoji et al. (2002) notes that ‘generally speakingmost members [of an open source community] arePassive Members. For example about 99 percent ofpeople who use Apache are Passive Users’. Similarlya user on the Mozilla Bugzilla comments:

’Reports from lots of users is unusual too;my usual rule of thumb is that only 10% ofusers have any idea what newsgroups are(and most of them lurk 90% of the time),and that much less than 1% of even Mozillausers ever file a bug. That would mean wedon’t really ever hear from 90% of users,unless we make some effort to reach them.’http://bugzilla.mozilla.org/show bug.cgi?id= 89907#c14.

Widening participation in bug reporting willrequire lowering effort, cultural, technical andusability barriers. Given the complexity and sub-jective nature of usability bugs and the need for

wider participation as compared to addressingconventional functionality bugs, we suspect thatthis usability system should be distinct from afunctionality-centric system such as Bugzilla andshould contain features that can address issues ofdetection and consolidation.

4.1.1. DetectionTo increase the likelihood of end users and usabilityexperts reporting usability problems, we need away of reporting usability bugs that is substantiallyeasier than Bugzilla. Bugzilla requires registrationand a careful following of a complex set ofnorms in order to ensure that bugs are reportedwith sufficient information to ensure replicationand to minimize duplication. This imposes asignificant burden on bug reporters, and indeedis not necessarily ideal even for current users anduses of Bugzilla. Previously we have consideredways in which crash reporting techniques could beadapted and extended to usability bug reporting(Nichols et al. 2003). These could vary in the degreeof participation required of a reported, from avery basic ‘I’m confused’ message accompaniedby an automatically determined system state at themoment of declared confusion, through to a user-provided description of the nature of the details oftheir confusion. The key twin concerns for wideningparticipation in reporting are ease of use (so thatit does not distract too much from the regular useof the software), and addressing privacy concernsabout automatically collected system state data (sothat the user is comfortable that this does not includeany confidential information).

4.1.2. ConsolidationGiven the aim of widening the pool of potentialbug reporters, any tool will need to deal withadditional challenges: coping with (hopefully) avery large number of reports – including manyduplicates, coping with erroneous, ambiguous andunhelpful reports and classifying and consolidatingthese reports. Large numbers can help designersto get a sense of the relative priorities of whichbugs to fix first, a critical piece of informationgiven limited resources. In this sense, duplicationof usability bugs (once consolidated) is valuable,whereas duplication of functionality bugs is oftenconsidered a waste of time and effort. Bugzillaaddresses consolidation by expecting bug reportersto carefully search to see if their bug already exists


154


in the database, and only if so, to add their reportto it, provided it adds to the overall understandingof the problem.

The complexity of discussions in the MozillaBugzilla has led to the development of a newtool, Hendrix (Hendrix 2004). Hendrix is an exper-imental and lightweight means for users to pro-vide feedback for cases where the use of Bugzillawould be onerous or inappropriate. It makes use ofpre-existing newsgroups used to discuss possibleextensions and bugs in Mozilla, separate from theformal Bugzilla reporting mechanism. Thus it sep-arates out detection functionality and makes thateven simpler for an end user to participate, whileusing pre-existing mechanisms for consolidation,analysis, design and consensus.

4.2. Analysis

Knowing that a particular interface element cancause confusion is not the same as knowing why.This can be achieved by careful analysis, by gen-erating hypotheses, by comparing the interface towhat is known about other systems or underly-ing HCI theory, by conducting user studies or byengaging the initial reporter in a more detaileddiscussion. In all cases analysis is likely to be dis-cursive, needing to address multiple alternativecandidate explanations and disagreements. Con-ventional bulletin-board functionality is likely to beimportant, but supplemented by visual aids suchas annotated screenshots to illustrate points underdiscussion. These must be sufficiently fast and easyto use so that they do not get in the way of thediscussion.

Some preliminary bug reports and analyses willbe controversial. Developers may just not believethat a particular design element really does causeconfusion, or if it does, for the reasons suggested,or that a proposed solution really is faster tolearn or easier to use and does not cause moreproblems than it solves. In such circumstances,additional data might be needed, either by passivelywaiting for more bug reports to appear, or byactively undertaking usability tests. Remote usertesting has been proposed as one possible approach(Hammontree and Weiler 1994). Elsewhere (Nicholsand Twidale 2003) we have advocated creating aninfrastructure for very small-scale usability studies,even as small as one user for 10 minutes (Martyand Twidale 2005). With a distributed team of

volunteers and appropriate coordination, this canlead to aggregated results sufficient to informongoing development.

4.2.1. DesignOnce the underlying cause(s) of the confusion hasbeen determined by careful analysis, it is possibleto propose design fixes to eliminate or amelioratethe problem. Of course care has to be taken thatthe proposed fix does not introduce new, worseproblems, or cause a reversion to an earlier prob-lem that the former fix addressed. Team-based peerreview, informed by a suitable design rationale ofearlier and ongoing problems and goals can helphere, just as they do in optimizing OSS functional-ity and reliability. The additional complexity is theimportance of consistency in interface design as amajor element of overall usability. Hence local fixesalso have to be considered for larger global effects.In coding functionality, much effort is devoted tominimising or eliminating global effects, and is akey software engineering design principle. Unfor-tunately, the global effects of interface consistencyare critical and cannot be eliminated.

4.2.2. Achieving Consensus (or At Least Making aCoherent Ruling)There may be considerable disagreement about therelative importance or severity of a usability bug,whether it is a bug at all, its underlying causes,and the most appropriate design fix. Nevertheless,some consensus must be achieved if progress is tobe made. Otherwise a failure to achieve consensuscan leave a usability bug languishing un-addressedsometimes for years (Twidale and Nichols 2005). Ifconsensus is not possible a ruling must be made.One way to achieve this more quickly is to takethe design work and discussion offline. This canbe done either in face-to-face meetings, or viasynchronous technologies such as phone or instantmessaging. In the absence of such mechanismsdebates can just run on, or until one side runsout of enthusiasm for continuing the debate. InGNOME the HIG can provide an external sourceof authority to help a group of developers achievea consensus; observations of usability bugs in theMozilla Bugzilla suggest that the lack of a well-used set of interface guidelines may contribute toextended discussions.


155


4.3. (Re)Design Work

Although there is no specific usability design infras-tructure in place analogous to the functionality-centric infrastructures such as CVS, SourceForgeand Bugzilla, we do see informal uses that havethe form of workarounds, appropriations and eventhe inspirations for ‘informalisms’ (Scacchi 2002) fora specification of a more usability-centric supportstructure. It is quite likely that a usability infras-tructure will evolve, just as one did for functionalitybugs. Such an evolution is likely to include compo-nents, or draw inspiration, from activities such asthose we note below.

4.3.1. Moving Design Work ElsewhereWe find mentions in the Bugzilla archive of designwork, particularly interface design work, occurring‘elsewhere’ than within the postings of Bugzilla.Possible reasons include:

• A desire to minimize the acknowledged com-plexity of Bugzilla by removing certain designdebates until a solution can be devised that canbe returned for public review and critique.

• A wish to use Bugzilla solely for more formalreview.

• The remote asynchronous text-centric natureof Bugzilla, which is less than ideal for thebrainstorming, design and debate of candidatesolutions.

Consequently, we find references to work occur-ring in face-to-face meetings, use of Mozilla news-groups, Internet Relay chats, emails, instant mes-saging, etc. (von Hippel and von Krogh 2003). Eachof these different venues in some way addressesone or more of the problems with Bugzilla for sup-porting interface analysis and design noted above.Unfortunately, although efficient, this can mean thatthe record of how the design evolved and how thedecisions were made is lost, unfortunate both forresearchers like us and more importantly for futurere-design work.

4.4. Design-by-Blog: A New Kind of Off-loadedDistributed Design Medium

One kind of innovative technological appropriationthat achieves the aim of off-loading interface designfrom the conventional bug reporting and revisioninfrastructure as embodied by Bugzilla is the use of

blogs. Blogs are used as supplements to Bugzilla-like systems, rather than as replacements, for thereasons outlined above.

Many open source projects have blogs to dis-seminate news about their progress. Individualbloggers have many motivations for writing includ-ing ‘providing commentary and opinions’ and‘expressing deeply felt emotions’ (Nardi et al. 2004),but from a project’s perspective they represent aneasy, standards-based, networked mechanism formaintaining communication across a distributedcommunity. As the technology of blogs is largelycontent-neutral, it has been a small step for projectsto supplement project news items with design andcode-oriented posts.

Typically a design-oriented blog entry containsa problem description, information about the goalsof the design and any constraints or design crite-ria, comments on earlier designs, etc. followed byscreenshots of the proposed design. This in turncan be followed by text describing the design, itsrationale and any particularly troubling or effectivecomponents. Further re-designs and rationales fol-low, often with a pattern of alternating screenshotsand textual commentary and rationales. Addition-ally, people other than the blog author will postcomments on the evolving design. Of course, fol-lowing the reverse chronological order of blogs, theevolving design has to be understood by reading upentries rather than down.

We illustrate how blog-based interface designcan be collaborative, transparent and distributedwith an extended example from the blog ofthe groupware client Evolution. The blog mixesseveral types of information including project news,meeting minutes, requests for help and proposeddesign changes; Figure 3 shows three (re)designs ofthe alarm dialog.

The design in Figure 3a was posted to theEvolution blog on 18 November 2004 with thefollowing text:

New Alarm DialogWorked on creating a mock-up for a re-designof the alarm notification dialog in Evolutiontoday, and after a few revisions, Ben and Iagreed on the image below. I’m working onthe code to get this implemented and into CVS,so it should be popping up on your desktopsoon.


156


(a) (b)

(c)

Figure 3. Incremental re-design of the Evolution alarm dialog – from the Evolution blog (formerly at http://codeblogs.ximian.com/blogs/evolution)

At the time of first writing (March 2005) his blogentry had 189 attached comments – we estimate that16 comments were ‘real’ and the reminder were‘comment spam’. Sample quotes from the ‘real’comments include: ‘In Gnome, icons usually go onthe left side of alert windows’, ‘what happens whenI have more than one reminder? Will I have multipledialogs . . .?’ and ‘I still feel that the basic Gnomealarm + calendar functionality shouldn’t dependon a whole email app’. Thus, comments rangedfrom low-level widget placement through possiblescenarios to inter-application dependencies. Thenext day this text accompanied the posting ofFigure 3b:

New Alarm Dialog AgainAfter my previous blog entry about the Alarmdialog mock-up, a lot of feedback was had, andI went off to implement a version. What I cur-rently have implemented is in the screenshotbelow. If anyone knows a standard localizableway to put the time/date information into a

single line, please, e-mail me. This version hasthe times as currently displayed by Evolution,as well as the appointment summary in bold,instead of just an ‘‘Appointment’’ label, andsingle-click snoozing

This entry’s comments included ‘I don’t seem tounderstand when the alarm is set to go off’, ‘thankgod someone is finally doing something about thealarm dialog’ and ‘I do agree with the commentabove though that said the title and time should belarger.’

Finally, this text accompanied the design shownin Figure 3c on 23 November 2004:

New Alarm Dialog in CVSFinally, the new alarm dialog is in CVS. Ascreenshot of what was committed is below.The icon was moved to the left side, andthe secondary text/widgets were re-aligned tomatch the HIG [Human Interface Guideline]2.0 Alert specs more precisely. This cleans up


157


the code quite a bit and gets rid of the HTMLwidget that we were using before.

This entry’s comments included ‘I hope (expect)the date/time convention follows the locale set-tings’ and a confirmation that it does. Previouswork (e.g. Herbsleb and Mockus 2003) has identi-fied the variety of tools used by globally distributedteams to support work activities; we believe thisis the first reported observation in the literature ofdesign occurring via a blog.

4.4.1. Analysing Design-by-BlogThere are several interesting elements about this‘design-by-blog’ episode, which seems to addressseveral of the problems outlined earlier, and whichwe believe can be generalized:

• It is an example of a small design team (inthis case presumably one person) presentingideas and reacting to feedback, allowing forthe exploration of a solution while maintainingsome design consistency. It is not really design-by-committee. This, and all the examples ofdesign-by-blog we know about are actuallyre-design – something exists already, it doesnot start with formal requirements, scenariosor anything similar – and it starts with code(Raymond 1998).

• It happens quickly, over a few days. This is inalmost all ways ideal, but it can mean that certainpeople with valuable insights may not see it intime to contribute.

• Design is more public and accessible thanBugzilla. Commenting does not require therather complex registration process of Bugzilla.Unfortunately, public discussions can be tar-geted with comment spam – this is a risk ofdoing things in public, and a side effect ofthe greater ease of participating compared toBugzilla. Blog use is also more public andtransparent than taking a Bugzilla design discus-sion offline to a face-to-face meeting or privateelectronic communication. The public interfaceinspection nature of interface design-by-blog canbe regarded as a HCI variant of code inspection(Lanubile et al. 2003; Lussier 2004).

• Unlike Bugzilla postings, the graphics are inlinein the blog postings, so are automatically themain object for discussion. However, the incor-poration of graphics is asymmetric. Comments

are all textual (apart from the HCI class re-designwho used a separate web server). Finding theright place for the user’s area of interest withinthe blog is also easier, but only because designblogs are far smaller, containing far fewer thanthe thousands of entries in Bugzilla that necessi-tate a highly complex search interface.

• Blogs as a genre have a conversational styleamenable to the more provisional, discursive,argumentative and iterative nature of inter-face design. This can be contrasted with thecomplexity-management focus on workflowprocessing in Bugzilla, which enables very largenumbers of bugs to be reported and anal-ysed, fixes proposed and solutions reviewed andapproved. The Evolution blog follows the typi-cal blog genre of many different items organizedtemporally. Different designs are explored at dif-ferent times in the blog, along with informationabout bug fixes, updates, news, help requests,code questions, etc. It is a far more informalorganization than the indexing of Bugzilla. Weassume (for now) that people used to the bloggenre do not have major difficulties with suchheterogeneous collections.

Blogs create a space to show the evolution ofdesign ideas in a way that does not clog up thealready complex bug tracking and workflow proce-dures of systems such as Bugzilla. This additionalspace allows an interleaving of design ideas, ratio-nales and critiques. As well as being very interestingfor researchers or students of HCI to track how adesign evolved, it can also help in the immedi-ate design process. Documenting design rationaleis acknowledged to be important: ‘designers rarelyproduce it in practice because of the substantial timecommitment required’ (Myers et al. 2000). Know-ing that there is a current audience reading one’srationale rather than a hypothetical future codemaintainer can be a great motivator, particularlywhen this audience responds with comments. Addi-tionally, alternate designs, their rationales and thereasons for their ultimate rejection, can be a valu-able element in achieving consensus about the finaldesign (Berkun 2000).

The various advantages and disadvantages ofthe current manifestations of design-by-blog canbe useful in informing both the development anduse of alternate technologies and infrastructuresand in developing technological and organizational


158


fixes to current problems. For example, it wouldnot be difficult for an OSS project to develop a blogharvester to accumulate, index and archive multipleblogs used in the design process.

Finally, we note that the distributed micro-leveldesign activity of blog use contrasts with the cen-tralized workflow management system of Bugzillain interesting ways. Both aim to address the prob-lem of complexity management, and each have theirown strengths and weaknesses as we have shown.It is perhaps ironic, and certainly contentious, tonote the parallels and distinctions between a cen-tralized (perhaps Cathedral-like?) structure such asBugzilla and a distributed (perhaps Bazaar-like?)structure such as many different design blogs (Ray-mond 1998).

5. DISCUSSION

Research in open source usability has the potentialto be valuable to all kinds of software development,not just OSS. Proprietary software developmentcompanies can be reluctant to allow access to theirdevelopment processes that may reveal competitiveadvantages or indeed weaknesses that they wouldprefer remained confidential. In contrast, the openaccess to the records of OSS projects creates almostan embarrassment of riches; for this preliminarywork we only scratched the surface of the dataavailable.

In this article we have explored the problemof talking about usability problems and designsolutions within the context of projects involvingvolunteers who are usually widely geographi-cally dispersed. Part of our aim was to showhow a clear articulation of problems can easilylead to ideas for design solutions. We suspectthat many readers will already have thought ofseveral socio-technical fixes while reading the prob-lem descriptions above. Technologically, a consid-eration of computer-supported cooperative work(CSCW) can help in designing, tailoring and intro-ducing functionality to better support distributedsoftware design that is mostly, but not entirely,asynchronous. Socially, the relative absence of HCIexperts in OSS projects can in part be compen-sated by the use of the distilled knowledge of HIGsand appeals to standard elements of HCI knowl-edge such as Fitt’s law. Better yet is to create anenvironment that encourages greater involvement

of HCI experts and usability analysis and designdiscussions. Any such interventions have to inte-grate with the highly successful practices of OSS inaddressing functionality bugs.

To help inform this ongoing work, it is useful tocompare OSS usability practices with two relatedactivities: other aspects of OSS development andindustry best practice in usability.

5.1. Comparing OSS Usability to Other OSSProcesses

Open source development relies on a variety oftools: compilers, mailing lists, build systems, ver-sion control systems, memory leak detectors, bugrepositories, etc. Many of these tools automatetasks where manual alternatives would be infea-sible. These tools are used by globally distributedvolunteer developers to coordinate their work andcommunicate their results. We have shown howusability activities relate to elements of the opensource infrastructure and are moving to populateothers such as blogs. In the light of this extensivetool use it is perhaps surprising that open sourceprojects appear not to be using any tools, beyondinterface construction tools such as Glade, to aidtheir usability evaluation. However, the HCI com-munity in general has not developed any significantautomated approaches for non-Web applications.

Hackers will generally (and sometimes reli-giously) only use OSS tools to develop OSScode, but few such tools exist to help develop-ers design guideline-compliant applications.Benson (2004)

We consider that research into usability-orientedevaluation tools could be extremely valuable forOSS communities that lack human usability exper-tise (Nichols and Twidale 2003).

The ‘many eyeballs’ approach to open source codeimprovement (Raymond 1998) relies on the righteyeballs observing the code and then taking somecorrective action. As such it has similarities with theconcept of globally distributed software inspection(Lanubile et al. 2003). Interfaces are slightly differentas users experience them directly, but many are notin the position to act on their observations. Althoughthe number of developers who have sufficient skillto ‘scratch their itch’ may be small, potentially allusers can contribute to an OSS community through


159


usability reporting. The remaining 90% of passivemembers (Nakakoji et al. 2002) represent a hugeuntapped resource for OSS: one that can be engagedby the right infrastructure. Developments such asdesign-by-blog and Hendrix show moves are beingmade in this direction.

5.2. Comparing OSS Usability to Industry ‘BestPractice’ Usability Methods

In terms of usability, there is a clear dividebetween the small OSS projects and the largeprojects that have company support (Benson et al.2004). The large supported projects are adoptingtechniques from mainstream HCI and appear tobe benefiting. The GNOME HIG is now a keyinfrastructural resource that guides developmentand streamlines discussions. A problem with theinterface can be legitimated by referral to the HIG.Any disagreement can really only be about whetherit does not in fact deviate from the HIG. By contrast,in projects lacking clearly specified guidelines, aproblem has to be noted as deviated from thereporter’s notion of usability norms, and both theproblem and the norm are subject to debate. Theconsequence seems to be the rather unstructureddiscussions that are noted in Section 3.

Most HCI techniques assume co-located workboth in usability analysis and interface design,whether it is formal usability tests, ethnography,think-aloud protocols, heuristic analyses, cognitivewalkthroughs, paper prototyping, participatorydesign or the use of toolkits. There is growinginterest in distributed software development ingeneral (e.g. Lanubile et al. 2003; Prikladnicki et al.2003). Similarly there is a growing interest in remoteusability testing as a way to address the problemsof assembling sufficient numbers of subjects, andcreating ecologically valid evaluations (Wichansky2000). Far less attention has to date been given todistributed usability design, and work in this areacould greatly benefit OSS.

6. CONCLUSION

Usability work in OSS projects needs and uses dif-ferent processes than work on functionality, fixingcrash bugs and improving reliability. From a small-scale study of two OSS projects already having asignificant concern for usability, we identified aset of issues that led to informal requirements for

a usability infrastructure. These include making iteasier to report usability problems, analyse themand propose re-designs. All of these have a discur-sive aspect – one does not just do something, onehas to talk about it with others at a distance. Inorder to achieve the larger solution of far widerparticipation in the conversations about usability, itis essential to have mechanisms that enable usabil-ity discussions to be much faster and easier toengage in.

Much can be learned from the ad hoc appropri-ations of available technologies by the currentlyrather small numbers of committed OSS usabilityanalysts and interface designers. These can giveclues both about how to propagate usability pro-cesses to other OSS projects and how to designtechnologies and methods that support usabilityprocesses more effectively. Usability activity is in astate of transition, with new ideas, approaches andtechnologies being explored, including design-by-blog and HIGs. These offer particular advantages inwidening participation and yet maintaining focus.That is critical in a part of software developmentwhere subjectivity and debate are inevitable andvaluable but which in excess can prevent efficientprogress. It may be that we are witnessing a tran-sition to a more systematic and well-supported setof software development processes for enhancingusability that parallel the earlier development ofmore systematic approaches for OSS functionalitydevelopment processes such as bug reporting andversion management by applications like Source-Forge and Bugzilla.

REFERENCES

Austin RD. 2001. The effects of time pressure on qualityin software development: an agency model. InformationSystems Research 12(2): 195–207.

Benson C. 2004. Meeting the challenge of open sourceusability. Interfaces: 9–12 Number 59, Autumn 2004.http://www.bcs-hci.org.uk/interfaces/interfaces59.pdf.

Benson C, Muller-Prove M, Mzourek J. 2004. Professionalusability in open source projects: GNOME, OpenOf-fice.org, NetBeans. Extended Abstracts of the Conferenceon Human Factors and Computing Systems. ACM Press:New York, 1083–1084.

Berkun S. 2000. Why good design comes from bad design,http://www.scottberkun.com/essays/essay08.htm,15 September 2005.


160


Feller J, Fitzgerald B. 2002. Understanding Open SourceSoftware Development. Addison-Wesley: London.

Frishberg N, Dirks AM, Benson C, Nickell S, Smith S.2002. Getting to know you: open source developmentmeets usability. Extended Abstracts of the Conference onHuman Factors in Computer Systems (CHI 2002). ACMPress: New York, 932–933.

German DM. 2003. The GNOME project: a case studyof open source, global software development. SoftwareProcess–Improvement and Practice 8(4): 201–205.

Gilb T. 1989. Deadline pressure: how to cope with shortdeadlines, low budgets and insufficient staffing levels.In Software Risk Management, Boehm B (ed.). IEEE Press:Piscataway, NJ.

Hammontree M, Weiler P, Nayak N. 1994. Remoteusability testing. interactions 1(3): 21–25.

Hars A, Ou S. 2002. Working for free? motivationsfor participating in open-source software projects.International Journal of Electronic Commerce 6(3):25–39.

Hendrix. 2004. Mozilla.org http://webtools.mozilla.org/hendrix/, 15 September 2005.

Herbsleb JD, Mockus A. 2003. An empirical studyof speed and communication in globally distributedsoftware development. IEEE Transactions on SoftwareEngineering 29(6): 481–494.

Lanubile F, Mallardo T, Calefato F. 2003. Tool supportfor geographically dispersed inspection teams. SoftwareProcess–Improvement and Practice 8(4): 217–231.

Lussier F. 2004. New tricks: how open source changed theway my team works. IEEE Software 21(1): 68–72.

Marty PF, Twidale MB. 2005. Usability@90mph: present-ing and evaluating a new, high-speed method for demon-strating user testing in front of an audience. First Monday10(7): http://firstmonday.org/issues/issue10 7/marty,15 September 2005.

Mockus A, Fielding RT, Herbsleb J. 2002. Two casestudies of open source software development: apacheand mozilla. ACM Transactions on Software Engineeringand Methodology 11(3): 309–346.

Myers KL, Zumel NB, Garcia P. 2000. Acquiringdesign rationale automatically. Artificial Intelligence forEngineering Design Analysis and Manufacturing 14(2):115–135.

Nakakoji K, Yamamoto Y, Nishinaka Y, Kishida K, Ye Y.2002. Evolution patterns of open-source software systemsand communities. Proceedings of the Workshop on Principles

of Software Evolution, International Conference on SoftwareEngineering. ACM Press: New York, 76–85.

Nardi BA, Schiano DJ, Gumbrecht M, Swartz L. 2004.Why we blog. Communications of the ACM 47(12): 41–46.

Nichols DM, Twidale MB. 2003. The usability of opensource software. First Monday 8(1): http://firstmonday.org/issues/issue8 1/nichols/, 15 September 2005.

Nichols DM, McKay D, Twidale MB. 2003. Participatoryusability: supporting proactive users, Proceedings of4th ACM SIGCHI NZ Symposium on Computer-HumanInteraction (CHINZ’03), SIGCHI: Dunedin, 63–68.

Nielsen J. 1993. Usability Engineering. Academic Press:Boston, MA.

O’Mahony S. 2003. Guarding the commons: howcommunity managed software projects protect theirwork. Research Policy 32(7): 1179–1198.

Prikladnicki R, Audy JLN, Evaristo R. 2003. Globalsoftware development in practice lessons learned.Software Process: Improvement and Practice 8(4): 267–281.

Raymond ES. 1998. The cathedral and the bazaar. FirstMonday 3(3): (March) at http://firstmonday.org/issues/issue3 3/raymond/, 15 September 2005.

Raymond ES. 1999. The revenge of the hackers. In OpenSources: Voices from the Open Source Revolution, Stone M,Ockman S, DiBona C (eds). O’Reilly and Associates:Sebastopol, CA, 207–219.

Sandusky RJ, Gasser L, Ripoche G. 2004. Bug reportnetworks: varieties, strategies, and impacts in aF/OSS development community. Proceedings of the 1stInternational Workshop on Mining Software Repositories(MSR 2004), Edinburgh, 80–84.

Scacchi W. 2004. Understanding the requirements fordeveloping open source software systems. IEE ProceedingsSoftware 148(1): 24–39.

Scacchi W. 2004. Free and open source developmentpractices in the game community. IEEE Software 21(1):59–66.

Smith S, Engen D, Mankoski A, Frishberg N, Ped-ersen N, Benson C. 2001. GNOME Usability StudyReport, Sun GNOME Human Computer Interac-tion (HCI), Menlo Park, CA. Sun Micro sys-tems, Inc.: at http://developer.gnome.org/projects/gup/ut1 report/report main.html, 15 September 2005.

Trudelle P. 2002. Shall we dance? Ten lessons learnedfrom netscape’s flirtation with open source UIdevelopment, presented at the Open Source Meets UsabilityWorkshop, Conference on Human Factors in ComputerSystems (CHI 2002), Minneapolis, MN, at http://www.iol.


161


ie/∼calum/chi2002/peter trudelle.txt, 15 September2005.

Twidale MB, Nichols DM. 2005. Exploring usabilitydiscussions in open source software, Proceedings of theThirty-Eighth Annual Hawaii International Conference onSystem Sciences (HICSS’05) (CD/ROM). IEEE ComputerSociety Press: Los Alamitos, CA, 8.

von Hippel E, von Krogh G. 2003. Open sourcesoftware and the ‘‘private-collective’’ innovation model.Organization Science 14(2): 209–223.

Wichansky AM. 2000. Usability testing in 2000 andbeyond. Ergonomics 43(7): 998–1006.

Yamauchi Y, Yokozawa M, Shinohara T, Ishida T. 2000.Collaboration with lean media: how open-source softwaresucceeds. Proceedings of the Conference on ComputerSupported Cooperative Work (CSCW’00). ACM Press: NewYork, 329–338.


162

Documents

Usability processes in open source projects