An Integrated Approach to Designing and Evaluating Collaborative Applications and Infrastructures

Computer Supported Cooperative Work10: 75–111, 2001.© 2001Kluwer Academic Publishers. Printed in the Netherlands.

75

An Integrated Approach to Designing andEvaluating Collaborative Applications andInfrastructures

PRASUN DEWANDepartment of Computer Science, University of North Carolina, Chapel Hill, NC 27599, USA(E-mail: [email protected])

(Received 8 January 1998)

Abstract. Collaborative systems include both general infrastructures and specific applications forsupporting collaboration. Because of the relative newness and complexity of these systems, it hasbeen unclear what approach should be used to design and evaluate them. Based on the lessons learnedfrom our work and that of others on collaborative systems, we have derived an integrated approachto researching collaborative applications and infrastructures. The approach can be described as asequence of steps: We decompose the functionality of collaboration systems into smaller functionsthat can be researched more-or-less independently. For each of these functions, we adopt general(system-independent) principles regarding the design and implementation of the function, identifycollaboration scenarios at multiple levels of abstraction, identify requirements based on the scenarios,adopt an interaction model to meet the requirements, realize the interaction model as a concrete userinterface, develop a logical architecture of the system, identify a physical architecture for placing thelogical components in a distributed system, develop infrastructure abstractions, use the abstractionsto implement applications, and perform lab studies, field experiments, and simulations to evaluate theinfrastructure and applications. As in other models with multiple phases, feedback from subsequentphases is used to modify the results from the previous phases. In this paper, we describe, illustrateand motivate this research plan.

1. Introduction

We use the term collaborative systems to include both general infrastructures(Dewan, 1993a) and specific applications (Dewan, 1993b) for supporting collab-oration. Thus, these systems include collaboration infrastructures such as XTV(Abdel-Wahab and Jeffay, 1994), Rendezvous (Hill et al., 1994), GroupKit(Roseman and Greenberg, 1996), Prospero (Dourish, 1995), MASSIVE (Green-halgh and Benford, 1995), COLA (Trevor et al., 1994), COAST (Schuckmann etal., 1996), Trellis (Stotts and Furuta, 1994), COCA (Li and Muntz, 1998), andSuite (Dewan and Choudhary, 1992); and collaborative applications such as theGROVE outline editor (Ellis et al., 1991), the PREP document editor (Neuwirth etal., 1992), the GroupDesign graphics editor (Karsenty et al., 1993), the FLECSEcollaborative software development environment (Dewan and Riedl, 1993), and the

76 PRASUN DEWAN

Scrutiny (Gintell et al., 1993) and EGRET (Johnson, 1992) collaborative softwareinspection applications.

While recently there have been several complex, apparently diverse, researchefforts spanning a variety of disciplines to develop and evaluate collaborativesystems, there has been relatively little work done in identifying the process ofcarrying out this research. Identifying a general research plan for designing andevaluating collaborative systems is an attractive idea, from at least three relatedpoints of view:• It can help identify existing research efforts that complement each other by

addressing different phases of the research plan. Conversely, it can allow newresearch efforts to focus on specific phases of this plan without having to carryout the whole, potentially complex, plan.

• It can help resolve debates between the various disciplines studying this arearegarding the role played by each discipline in fulfilling the plan.

• It can be used as a reference for the research efforts in this new and not sowell understood area, which are often asked to explicitly outline and justifytheir research plans.1

There is, of course, previous work in process models such as the spiral andwaterfall models, but these models are meant to describe steps in the develop-ment of arbitrary software, and are, thus, too general to satisfy the three objectivesabove. Based on the lessons learned from our work and that of others on collabo-rative systems, we have derived an integrated approach to developing collaborativeapplications and infrastructures that meets these objectives to a significant extent.Like the more general models, it is not intended asthe process for collaborationresearch, but instead a first-cut effort at identifyinga process for such research. Itbreaks down some of the general steps of the previous models into more specificsteps, which include decomposing the functionality of collaboration systems intosubfunctions that can be researched independently, adopting general (system-independent) principles regarding the design and implementation of the function,identifying collaboration scenarios at multiple degrees of abstraction, identifyinggeneral requirements based on the scenarios, adopting an interaction model tomeet the requirements, realizing the interaction model as a concrete user interface,developing a logical architecture of the system, identifying a physical architecturefor placing the logical components in a distributed system, developing softwareabstractions, implementing the abstractions in an infrastructure, implementing theapplications and performing lab studies, field experiments, and simulations to eval-uate the infrastructure and applications. As in the waterfall and spiral models,feedback from subsequent steps is used to determine the results from the previousphases. Not all steps are useful for every collaboration project; in fact, we do notknow of any project that includes all of them. Our goal, in this work, has been toidentify as many steps as possible together with the pros and cons of supportingthese steps so that it can be used to distinguish among and guide research efforts inthis area.

AN INTEGRATED APPROACH TO DESIGNING AND EVALUATING 77

In this paper, we describe, illustrate, and evaluate the various step in thisplan. We often omit the term collaborative in the terms we use such as collabo-rative applications, infrastructures, taxonomies, interaction models, abstractions,and architectures. For instance, we use the term “interaction model” to refer to a“collaborative interaction model”.

The rest of the paper is organized as follows. Section 2 gives an overview ofthe proposal research plan, identifying its steps. Section 3 is the main part of thepaper, describing, motivating, and illustrating the different phases of the researchplan. Finally, Section 4 presents conclusions and directions for further research.

A version of this paper addressing the evaluation component of the researchplan was published in a workshop on evaluating collaborative systems (Dewan,2000).

2. Research steps

Our work is related to previous work on the spiral and waterfall models. Asmentioned above, the spiral and waterfall models describes general approachesto carrying out software projects. The spiral model is a hierarchical model thatdivides software development into multiple steps represented by different cycles ofa spiral, and then divides each cycle into multiple stages represented by differentquadrants of the cycle. The output of each cycle is some version or portion ofthe final product, and the objects created in a cycle are allowed to depend onthose created in previous cycles. The model does not characterize the numberof cycles or the relationships among them. It does, however, characterize thestages in a cycle, which consist of identification of alternatives, evaluation of thesealternatives, development of parts/versions of the product, and evaluation of theproduct.

The waterfall model breaks down a project into following successive phases:requirements identification, design, coding, testing, and maintenance. The require-ments phase of the waterfall model combines the alternatives identification andevaluation stages of the spiral model. Similarly, the testing phase of the formercorresponds to the more general stage of evaluation in the latter. The waterfallmodel does not explicitly support the notion of creating new products based onthe previous one. However, it does support the more general notion of incrementaldevelopment by allowing earlier phases to be revisited based on feedback fromprevious phases. The feedback could be used to refine the existing product or builda new product.

Since collaborative systems are software systems, these two models apply alsoto the development and evaluation of these systems. However, these general modelsdo not give important collaboration-specific details regarding some of these phases.In particular, they do not address how requirements of collaborative systemsshould be identified or how these systems should be designed, implemented, andevaluated.

78 PRASUN DEWAN

Figure 1. Diagram of research plan.

Our research plan attempts to answer some of these questions for both collab-orative applications and infrastructures. It is described by the connected boxesshown in Figure 1, which represent different steps in the project. As in the water-fall model, each step produces output for and receives feedback from subsequentstep(s), which could be used, as in the spiral model, to determine the next product.In fact, we have included infrastructure steps in the process because we expectincremental development. None of the steps, however, is mandatory; moreover,output/feedback is not restricted to the immediately succeeding/preceding phases,and may bypass them to reach any of the succeeding/preceding steps.

The order in which these steps are taken depends on the way the plan is imple-mented. In a pure top-down implementation, the steps are taken top to bottom,while in a pure bottom-up implementation, they are taken in the reverse order. Asin the waterfall model, the level of a step is determined by how abstract it is –a higher-level step such as principles identification is more abstract than a lower-level step such as requirements identification. An arrow from the from the formerto the latter is considered as output and latter to the former as feedback.


The first eight boxes address steps that are common to the research of boththe infrastructure and applications. They include decomposition of the function-ality of a collaborative system into individual collaboration functions, adoptinggeneral (system-independent) principles regarding the design of the function,identification of the kinds of scenarios to be supported, identification of require-ments based on these scenarios and principles, adoption of an interaction modelsupporting these requirements, realizing the interaction model as a concrete userinterface, and developing a logical and physical architecture of the collaborativesystem.

At this point, it may be possible to identify general abstractions for automatingaspects of the user interface and logical and physical architectures, which are thenimplemented in an infrastructure. The next step is application development.

The subsequent boxes describe methods for evaluating the infrastructure andapplications. These include inspection of the designs to observe if they meet theirrequirements, simulations of existing collaborations systems, complete solutionsto standard problems, and self, lab, and field studies regarding usability of thecollaborative applications, programmability of the infrastructure, and performanceof the applications and infrastructure. As shown in the figure, some of these stepshave no dependence among them, and are thus expected to be carried out inparallel.

In the following sections, we motivate, explain, and illustrate the componentsof this research plan.

2.1. RELATIONSHIP BETWEEN APPLICATIONS AND INFRASTRUCTURE

RESEARCH

A collaborative application helps its users to perform some collaborative task;while a collaboration infrastructure automates part of the functionality of a setof collaborative applications. Both collaborative applications and infrastructuresmay be build on top of general purpose software infrastructure such as a data-bases system, user-interface tool, or distributed system (Dourish, 1998b). Thedifference between a collaboration infrastructure and the underlying softwareinfrastructure is that the former adds to the latter collaboration-specific (butapplication-independent) layers, often called a collaboration toolkit (Roseman andGreenberg, 1998), for implementing collaboration functions.

Several projects in collaborative systems, especially the early ones, havefocused either on applications or on infrastructures. For instance, shared windowsystem projects (Lantz et al., 1987) concentrated mainly on infrastructures forautomatically converting a single-user application to a collaborative application,without seriously identifying what the ideal design of the collaborative applicationshould be. Conversely, projects such as the Grove outline editor (Ellis et al., 1991)and the ICICLE code inspector (Brothers et al., 1990) addressed only the design

80 PRASUN DEWAN

of a specific collaborative application without considering how parts of the designscould be automatically supported in a collaboration infrastructure.

We argue that infrastructures and applications should be designed together inan integrated fashion, whenever possible, for two main reasons:

• Applications are required to evaluate the programmability of the infrastruc-ture. Moreover, they are necessary to identify the range of concrete scen-arios that infrastructures should support. Thus, it is important that specificapplications be considered in the design and evaluation of infrastructures.

• Conversely, infrastructures can make it easy to implement and change appli-cations. As mentioned before, collaborative applications can be far morecomplex than several other kinds of software systems because of the varietyand number of issues they must address. Moreover, they require more itera-tions of usability testing and reimplementation than single-user applicationssince (a) more users with diverse needs are involved, and (b) they arerelatively new and thus few well established principles exist for designingthem. Therefore, it is crucial that complex applications be implemented usinginfrastructures rather than from scratch.

The main goal of the early infrastructure projects was to demonstrate thewide applicability of collaborative user interfaces by automatically creatingsomecollaborative interface for any single-user application. The main goal of theearly application projects was to demonstrate that complex design issues must beresolved before collaborative interfaces are successful. As the general applicab-ility and complexity of these applications have become recognized, it has becomemore important for future projects to address collaborative systems in an integratedmanner by generalizing application-independent design concepts in collaborationinfrastructures and implementing applications using infrastructures.

The idea of integrated research of collaborative infrastructures and applica-tions is not new. In particular, the early Xerox Colab project included both a setof collaborative applications and a programming environment for implementingthem (Stefik et al., 1987). Recently, there have been more examples of integratedapplication-infrastructure development such as the GroupDraw-GroupKit (Green-berg et al., 1992; Roseman and Greenberg, 1996), Scrutiny-CB (Gintell et al.,1993; Kaplan et al., 1992) UARC-DistView (Clauer et al., 1994; Prakash and Shim,1994) FLECSE-Suite (Dewan and Choudhary, 1992; Dewan and Riedl, 1993) andthe MASSIVE (Benford et al., 1999; Benford et al., 1997; Bowers et al., 1996a;Bowers and Martin, 1999; Greenhalgh and Benford, 1995) projects.

To illustrate the usefulness of this approach, consider our experience with theintegrated development of the FLECSE (Dewan and Riedl, 1993) and Suite (Dewanand Choudhary, 1992) collaborative systems. Suite is a collaborative infrastructurethat automates the implementation of several collaboration functions includingcoupling (Dewan and Choudhary, 1995), access control (Dewan and Shen, 1998a,b), undo/redo (Choudhary and Dewan, 1995), merging (Munson and Dewan,


1994), and concurrency control (Munson and Dewan, 1996). It defines a widerange of useful collaboration policies for these functions and allows end-users todynamically change these policies as they change the nature of their collaboration.FLECSE (Flexible Environment for Collaborative Software Engineering) is a setof applications supporting collaboration in all phases of the software life cycleincluding design, coding, debugging, and testing. It consists of a collaborativesoftware inspector, program editor, debugger, testing application, and commandinterpreter. All of the FLECSE application have been implemented using Suite.

By using an infrastructure to build FLECSE, we have been able to easilyimplement and change a wide range of collaborative applications. Moreover,we have been able to offer in each of these applications the whole range ofcollaboration functions provided by Suite. Comparable- or bigger-sized softwareengineering projects that have attempted to build applications from scratch havefocused only on a single application, supported only synchronous coupling andprovided limited or no support for concurrency control, access control, undo/redoor merging (Brothers et al., 1990; Gintell et al., 1993). Part of the reason forthis is that often these projects were mimicking face-to-face meetings, which areheld synchronously without automated support for access control, undo/redo, ormerging. Our goal of a general infrastructure forced us to think of a wide range ofactivities (including synchronous and asynchronous collaboration), and we imple-mented the union of the collaboration functions and policies we envisioned fordifferent applications. Once this set of choices was available for each applica-tion, we were able to find uses for options we did not originally consider forthe application. For instance, we were able to find uses for both synchronousand asynchronous software inspections, though we had originally only consideredsynchronous inspections.

Conversely, by developing FLECSE using Suite, we have been able to evaluatethe system and redesign it based on the feedback received from the programmersand users of FLECSE. Without thinking of a concrete set of applications, we wouldnot have been able to identify a wide range of collaboration scenarios, which werenecessary to determine the range of policies supported by Suite.

It is important to note that integrated application-infrastructure development iscurrently not always possible for all applications. For instance, Suite and othercollaboration infrastructures we know of do not currently support automatic gener-ation of arbitrary graphics applications. As a result, such applications must bedeveloped currently from scratch until further research can be done to creategeneral infrastructures for them.

Abstracting application-specific features into application-independent infra-structure concepts is difficult and may not be as efficient. Thus, it may be omittedfor simple applications or the initial passes through the research plan when one ormore concrete applications are built from scratch to understand what is needed inthe infrastructure. This bottom-up approach to designing infrastructure components

82 PRASUN DEWAN

was taken in Chung’s work on latecomer accommodation. He first developed XTV-specific latecomer support (Chung et al., 1993) and then later abstracted this workto a generic latecomer module for arbitrary application (Chung et al., 1998). Inthe bottom-up approach, we may not revisit the infrastructure step directly afteran application-feature is implemented from scratch. If the application is a proto-type, then we may revisit the scenarios and requirements steps to decide if it isworthwhile to abstract the feature into an infrastructure.

2.2. FUNCTION DECOMPOSITION

In several traditional areas, the task of researching a complex software system ismade tractable by subdividing the function of the system into subfunctions. Forinstance, an operating system is typically divided into several subfunctions suchas process management, interprocess communication, memory management, andfile management, and each of these functions is designed and evaluated (more orless) independently. The subfunctions are not always completely independent sincethe design of one sometimes depends on the design of another. For instance, thedesign of process management depends on the interface provided by the memorymanagement component. However, these components are loosely related in thatthe resolution of complex issues in the design of one function does not influencethe resolution of issues in the design of another function.

Compared to other software systems, collaborative systems can be far morecomplex, since they may include complex designs from multiple areas such as user-interfaces, databases, and distributed systems (Dewan and Shen, 1998a; Dourish,1998b). For instance, the GroupDraw shared whiteboard (Roseman and Green-berg, 1996) provides gestalt viewers and multiuser-scrollbars to support awareness,a combination of pessimistic and optimistic transactions to support concurrencycontrol, and a replicated architecture to support interactive feedback to local events.The task of researching such systems can be made more tractable by using theapproach of function decomposition.

Not all projects, especially the early ones such as Xerox’s Colab project (Stefiket al., 1987) have taken this approach. Since the goal of most of these projectswas to do the initial demonstration and evaluation of these systems, they did notconsider in-depth all of the collaboration functions known today. The experienceand systems provided by these projects has provided a basis for now taking afunction decomposition approach.

Several recent projects have implicitly taken this approach. For instance,Prakash has focussed in-depth on undo (Prakash and Knister, 1994) and sessioncapture/replay (Manohar and Prakash, 1995), Greenberg has focussed in-depth onconcurrency control (Greenberg and Marwood, 1994), and awareness (Gutwin andGreenberg, 1999), and the PREP project has focussed on coupling (Neuwirth etal., 1994) and diffing (Neuwirth et al., 1992). Based on these projects and our own


project, we have decomposed the functionality of a collaborative system into thefollowing functions (Dewan et al., 1994):• single-user interaction – determines the effect of users’ commands on their

own displays;• coupling – determines the effect of users’ commands on the displays of other

users;• undo – allows users to cancel/repeat previous actions;• diffing – finds the differences in independent versions of an object;• merging – combines independent versions into a single object;• access control – determines the operations a user is authorized to perform;• concurrency control – prevents inconsistent concurrent authorized interac-

tions;• process/workflow management – prevents execution of authorized commands

that are inconsistent with the current process/workflow step and also automat-ically enacts commands according to the current process/workflow step.

• awareness – makes users aware of “out of band” activities of their collab-orators, that is, activities not deducible from the application feedback theyreceive from coupling, concurrency control, and other functions above.

• session management – allows users to start/stop interaction with a collabora-tive application.

Each of these functions has a semantics and user-interface component. Forinstance, the concurrency control module may have a semantic component thatlocks shared objects and a user- interface component that provides users withcommands to invoke the lock operation.

Olson et al. (1993) similarly decompose this functionality into linking (corre-sponds to coupling in our model), awareness, and undo. Ellis (1994) uses a differentset of dimensions, decomposing the functionality into ontological, coordination,and user-interface models. The ontological model captures the semantics of single-user interaction and access control, the coordination model includes the semanticsof concurrency control and process/workflow enactment, while the user-interfacecomponent determines the user-interface of single-user interaction, coupling, andawareness.

Thus, as we can see above, the way(s) in which collaboration functionalityshould be decomposed is still a matter of research. Moreover, while an initialdecomposition phase is necessary to determine the focus of a project, this decom-position can, and does, change based on the results of later phases. For instance,our initial decomposition (Dewan et al., 1994) did not include merging, whichwas added later when we faced the need for merging different versions createdby asynchronous coupling.

Though it is possible to individually research the various collaboration func-tions, there are dependencies among the points in these design spaces, that is,high-level goals (such as real-time collaboration) can be met by only certain

84 PRASUN DEWAN

combinations of these points. For instance, research has shown that WYSIWIS(What You See Is What I See) (Stefik et al., 1987) coupling and no concurrencycontrol go together (Ellis et al., 1991), that is, it is possible and desirable to doWYSIWIS collaboration without concurrency control since (a) users can them-selves prevent inconsistent interaction when they see all remote actions and (b) nosystem resources are used to check these inconsistencies. Therefore, it is necessaryto have a phase that integrates the research of the various functions researched inde-pendently. For instance, we initially developed our coupling, concurrency control,and merging schemes independently; and later developed a scheme for automatic-ally generating part of the concurrency control semantics from merge and couplingsemantics (Munson and Dewan, 1996).

The decomposition approach, of course, may be too heavyweight for simpleapplications providing limited functionality. For instance, it may be overkill for aninternet chat tool, which does not need concurrency control, access control, multi-user undo, or workflow support, and requires very simple forms of coupling andawareness support. The approach in general, and the particular function decom-positions mentioned above, are more appropriate for collaborations involvingmanipulations to a shared object.

2.3. SCENARIOS

A scenario is a sequence of interactions with an application. In the initial phases ofa project, scenarios are hypothetical; once the application is implemented and used,they reflect actual interactions with the application. It is important to use scenariosto derive the requirements of a collaboration system to understand and explain thecomplex interactions possible among the users of these systems.

A scenario can beapplication-specificor application-independentbased onwhether it is described in terms that are specific to a particular application. Thefollowing is an example of an application-specific scenario from our paper onFLECSE (Dewan and Riedl, 1993), which describes interaction in terms that arespecific to the FLECSE software inspection application:

Structured asynchronous annotations allow Alice and Chou to make their argu-ments privately without bothering each other. Once they are ready to presenttheir arguments, they commit their annotations to make them visible to otherprogrammers. At this point, the collaborative software inspection tool alsopresents an integrated list of faults, sorted by estimated severity.

Here is the abstraction of the scenario described in application-independent terms:

Structured asynchronous editing of data structures allows users to make theirchanges privately without bothering each other. Once they are ready to sharetheir work, they commit their changes to make them visible to other users. Atthis point, the collaborative application responds to the users’ input by updatingthe display.


It is important to identify both kinds of scenarios. Application-specific scenariosare necessary for identifying the requirements of specific collaborative applica-tions. They are also necessary for deriving the general application-independentscenarios. Furthermore, they make more convincing cases for some functionalitysince they show specific benefits of the functionality. For instance, the application-specific scenario above makes a better case of asynchronous coupling by showingthe specific benefits of it in software inspection.

Application-independent scenarios, on the other hand, are necessary to derivethe requirements of infrastructures. Moreover, they form a more succinct repre-sentation of user interactions since a single application-independent scenario candescribe the entire range of application-specific instances of it. For instance, theapplication-independent interaction above represents not only the asynchronousediting of annotations but also design documents, program code, and spreadsheets.Furthermore, they can be used to generate new application-specific scenarios notconsidered before. For instance, the application-independent scenario above can beused to generate the following spreadsheet-specific scenario:

Structured asynchronous editing of the spreadsheet cells allow Alice and Chouto work privately without bothering each other. Once they are ready to sharetheir changes, they commit the edited cells to make them visible to otherusers. At this point, the collaborative spreadsheet recomputes the values of thedependent cells and displays these values to all users.

Thus an application-specific scenario can be used to derive an application-independent scenario, which in turn can be used to derive additional application-independent scenarios.

How should the initial application-independent scenario be identified? Oneapproach is to use scenarios of groups carrying out some collaborative activityto derive analogous scenarios for applications that provide computer support forthat activity. Kaplan refers to these scenarios as narratives (Kaplan et al., 1994). Toillustrate, in the design of FLECSE, real-world models of physical inspection meet-ings were used to derive the scenarios for computer-supported inspection meetings(Mashayekhi et al., 1993). For instance, in the real-world models, the inspectionmeeting begins with physical distribution of the document to be inspected. Inan analogous scenario for the inspection application, the meeting is started bydisplaying the document on the screens of the users of the application.

Real-world scenarios allow collaborative systems to mimic face-to-face meet-ings by giving their distributed users the feeling of ‘being there’. However, oneof the main advantages of collaborative systems is that they make it possible togo ‘beyond being there’ (Hollan and Stornetta, 1992) by supporting collaborationmodes not possible or practical in physical meetings. For instance, an inspec-tion application can allow a user to share a comment synchronously with someusers and asynchronously with others. To support these collaboration modes, it isnecessary to do ‘thought experiments’ to derive useful scenarios of collaboration

86 PRASUN DEWAN

not grounded in physical meetings. It is important to note that scenarios derivedfrom both actual and thought experiments are hypothetical until they are actuallycreated in some interaction with a collaborative system. Therefore, it is particularlyimportant to test, after the system is developed and used, whether the latter everbecome actual scenarios (Bowerset al., 1996a, b).

Our classification of scenarios into application-dependent and application-independent recognizes that the same activity can be described in concrete orabstract terms. This idea is also used in a report of the DARPA (Defense AdvancedResearch Projects Agency) evaluation group (Cugini et al., 1996), which uses fourlevels of abstraction in dividing scenarios into (in increasing order of specificity)requirement-level, capability-level, service-level, and technology-level scenarios.The requirement-level (not to be confused with the requirements discussed in thenext section) describes the scenario in terms of generic collaborative tasks such asplanning, brainstorming, and decision-making. The capability-level describes it interms of generic functions used such as displaying, viewing, querying, and linking.The service-level specifies the kind of applications used such as whiteboard andemail. Finally, the technology-level identifies the specific application used suchas Unix mail. Our application-independent and application-dependent scenarioscorrespond roughly to capability-level and service-level scenarios, respectively.The main difference is that we assume that the capability-level is defined formallyto the extent required to automate it in an infrastructure.

Scenarios can be complete or partial based on whether they include only certainkinds of interactions. They correspond to the “thick” and “focussed” narrativesdefined by Kaplan et al. (1994). To illustrate, a partial scenario may includeonly coupling interactions while the corresponding complete scenario would alsoinclude steps related to undo and other collaboration functions. Partial scenariosthat include interactions relevant to a particular collaboration function are crucialfor designing the function independently. For instance, we initially considered onlycoupling-specific scenarios to derive the design of the coupling model (Dewanand Choudhary, 1995) and later considered access-specific scenarios to derive thedesign of the access model (Dewan and Shen, 1998b). Complete scenarios, onthe other hand, are necessary for identifying inconsistencies between the designsof different collaboration functions. For instance, a complete scenario is needed toidentify the conflict mentioned above between WYSIWIS and concurrency control.Complete scenarios can later become standard problems (discussed later) used toevaluate collaborative systems.

Who should identify the scenarios? The obvious approach is to ask the intendedusers to think of ways in which their work could be automated (White, 1994).However, the users may not have enough insight into CSCW to do so. If the rapid-prototyping approach (Andriole, 1994) to requirements gathering is used, then it ispossible to use the designers’ imagination and experience with other systems in thefirst pass and involve the end users once the first prototype is developed. This was


the approach used in our project. A third alternative, which requires more effortbut offers the advantages of both of the above two approaches, is to let someoneknowledgeable about the possibilities of CSCW observe how the intended userscollaborate with each other, and based on these observations, derive scenarios inwhich the computer is used to aid the collaboration. There have been a significantnumber of projects that have used this approach in various domains such as airtraffic control (Bentley et al., 1992b), fashion (Pycock and Bowers, 1996), andambulance control (Bowers and Martin, 1999).

2.4. PRINCIPLES

Scenario identification is one way to derive the requirements of a system. Anotherway is to derive them from general principles, that is, rules regarding designand implementation of a collaboration function that are independent of specificscenarios, models, abstractions, and architectures, and thus can be applied toarbitrary projects in collaboration applications and infrastructures. If this step isbeing taken for the first time in the research plan, these would be products ofprevious research efforts; otherwise they may have been discovered in previousiterations of plan. While there are currently no standard accepted collabora-tion principles, some projects have attempted to identify and base their workon them. For instance, several systems such as Prospero (Dourish, 1998a) andConversation Builder (Kaplan et al., 1992) are based on the principles of situatedaction (Suchman, 1987), which recognizes that collaborators work opportunist-ically (Beck and Belloti, 1993; Harper et al., 1991; Heath and Luff, 1991), andtherefore requires that systems provide flexible mechanisms that allow dynamicchanges in the collaboration style. Similarly, several collaborative access-controlmechanisms (Dewan and Shen, 1998b; Sikkel, 1997; Trevor et al., 1994) arebased on the classic need-to-know principle (Saltzer, 1974), which says that usersshould not be able to gain access to information they do not need to perform theirtask.

Our work has suggested several initial principles regarding the architecture(Dewan, 1992) and collaboration functions (Dewan et al., 1994). The GroupKitproject has similarly identified principles regarding concurrency control (Green-berg and Marwood, 1994) and awareness (Gutwin and Greenberg, 1999). The truthof many of these principles is self evident once they have been stated, but several ofthem have not been followed in the design of collaboration systems. An example isthe need-to-tell coupling principle (Dewan et al., 1994), which requires that thecoupling function not transmit any information about a user’s local workspaceto another user that the former does not need to transmit to the latter. The prin-ciple is not followed, for instance, when a collaboration system forces sharing ofuncommitted results but its users want sharing of only committed results.

88 PRASUN DEWAN

2.5. REQUIREMENTS

Three important requirements followed in the design of traditional softwaresystems are the requirements of efficiency, flexibility, and ease-of-specification.The efficiency requirement implies that storage costs and response times contrib-uted by the system be kept low. It is the main requirement in the design of systemssuch as compilers and operating system kernels that address implementation ofexisting, well-accepted designs.

The other two requirements have more abstract, subjective and system-dependent semantics. The flexibility requirement implies that users of the serviceimplemented by the system find that it meets their needs; otherwise they would beforced to either use another system or accept less than ideal semantics for it. Fora collaborative application, the flexibility requirement implies that the applicationbe able to support the kinds of collaboration scenarios desired by its end users. Forinstance, for a software inspection tool, it implies that the tool support the coupling(synchronous/asynchronous) desired by its users. For a collaborative infrastructure,the flexibility requirement implies that programmers be able use the infrastructureto develop the kinds of collaborative applications they are asked to implement.

A flexible system must offer its users primitives to exercise choices. The ease ofspecification of a system measures the effort required to use these primitives. Thus,in the case of the software inspection tool, it would measure the effort requires tospecify the coupling, and in the case of an infrastructure used to develop the tool,it would measure the effort required to implement the tool. A system requiring lessspecification effort than another is often termed higher-level than the other.

The efficiency, flexibility, and ease-of-specification of an application is relatedto the corresponding characteristics of the infrastructure used to develop it. Anapplication cannot be efficient if its runtime components added by the infra-structure are not also efficient. Moreover, an application cannot support anapplication-specific scenario if the infrastructure does not allow the correspondingapplication-independent scenario. For instance, a software inspection applicationcannot support asynchronous coupling if the infrastructure does not allow suchcoupling to be implemented. The ease-of-specification requirements of the twokinds of systems are also related, though in a more subtle way. An infrastructurethat offers high-level runtime, user-settable parameters to its programmers relievesthem from the task of implementing these parameters. For instance, the Suite infra-structure offers high-level runtime parameters for switching between synchronousand asynchronous coupling, which are available to users of all Suite applicationssuch as the FLECSE software inspector. As a result, from the point of view ofthe users of the inspector, the application supports ease-of-specification, since theycan easily enter the parameter values. Similarly, from the point of view of program-mers of the application, Suite supports ease-of-specification, since the parametersare automatically implemented for them – all they have to do is specify anyapplication-specific default values of these parameters. These three characteristics


of an application are not, of course, implied by the corresponding characteristicsof the infrastructure used to develop it: an efficient/flexible/low-level infrastructuremay be used to implement an inefficient/inflexible/high-level application. Thus, itis important to evaluate both kinds of systems.

Besides these three requirements, other pragmatic requirements must alsobe considered. In particular, an important requirement for several collaborationsystems is reuse of some class of existing single-user applications. In its mostextreme form, this requirements implies that the single-user applications be sharedby multiple users without any changes to them. Several collaboration infra-structures have been designed to support this requirement. For instance, XTV(Abdel-Wahab and Jeffay, 1994) and Shared X (Garfinkel et al., 1994) have beendesigned to allow sharing of existing single-user X applications and Suite has beendesigned to allow sharing of existing single-user Suite applications.

Some of these requirements can conflict with each other. In particular, theease-of-specification and reuse requirements can conflict with the flexibilityrequirement. The ease-of-specification requirement is often met by making certainassumptions in the infrastructure, which conflicts with the flexibility requirement.For instance, a system such as Suite that allows a programmer to specify thecoupling by setting some parameters is higher-level but less flexible than a systemsuch as GroupKit that allows the programmer to write arbitrary code to do so.Similarly, the extreme case of the reuse requirement implies that existing single-user applications be shared without adding any collaboration-aware code in them,which implies that these applications do not have the flexibility to determine howtheir collaboration functions are carried out. For instance, shared window systemfix WYSIWIS coupling for all applications, which are entirely collaboration-unaware. As a result, end-users and programmers of these applications are freedfrom the task of specifying the coupling but are also not given the option oflooser couplings. Therefore, designers of collaboration systems must often tradeoffthe flexibility requirement for the ease-of-specification and reuse requirements.A more important conclusion is that it is often difficult to prove that a systemis “better” than another since the two will typically make different tradeoffs inmeeting conflicting requirements. Often the challenge in the design of collabora-tion systems is finding new, useful ways to make these tradeoffs. For instance, mostof the infrastructure research in collaboration systems has involved finding newcompromises between the complete automation offered by a shared window systemand the complete flexibility offered by a general-purpose distributed programmingenvironment. Such a compromise must be justified by showing that it supportssome set of tasks (collaboration scenarios/applications) that would require moreeffort in all other systems that are flexible enough to support it. For instance,only a shared X window system can support WYSIWIS sharing of collaboration-transparent single-user X applications. Similarly, Suite supports a set of coup-

90 PRASUN DEWAN

ling polities that would require more effort to specify in all other systems thatsupported it.

The definitions of ease-of-specification, reuse, and efficiency requirements areindependent of the collaboration function and do not need further qualification. Onthe other hand, the meaning of the flexibility requirement depends on the exactfunction being considered, since it must specify a range of tasks to be supported,which are dependent on the function. As a result, the flexibility requirement mustbe decomposed into simpler function-specific requirements. For instance, in thecase of Suite, a coupling-specific requirement is that it should be possible fora user to be coupled differently with different users and a concurrency-controlspecific requirement is that variable-grained locking granularity should be support.Together, all the function-specific requirements of Suite (Dewan et al., 1994) defineits top-level flexibility requirement.

Like all the other steps in the research plan, this step is also optional. However,few successful systems projects omit this step, since it helps focus the project,succinctly indicates what is unique about its goals, and most importantly, providesa basis for evaluating the system produced by the project. In fact, in this author’sexperience, the main difference between a paper (about some system) that isaccepted for publication and one that is not is whether the paper explicitly showsthat the system satisfies a unique set of requirements.

2.6. INTERACTION MODEL

An interaction model describes a user-interface from a high-level point of view.Once the scenarios, principles and requirements have been identified, the nextstep is to design or adopt an interaction model for the system. If the system isan application, the interaction model is an abstraction of its user-interface; while ifit is an infrastructure, the interaction model is an abstraction of the user-interfacesof the applications supported by it.

Differentiating between abstract models and concrete languages/user-interfacesis common in other fields. For instance, programming models such as logic, func-tional, and object-oriented programming are distinguished from the programminglanguages such as LISP, Prolog, and Smalltalk that implement them. Similarly,database models such as the relational, navigational, and object-oriented modelsare distinguished from the database schema definition languages that supportthem.

An example of a collaborative interaction model is therooms model(Ensor etal., 1988), used explicitly in the design of Rapport (Ensor et al., 1988) and othershared window systems; and also supported by GroupKit (Roseman and Green-berg, 1996), DistView (Prakash and Shim, 1994), and MUDs (Curtis, 1992). Themodel is based on the real-world “rooms” model, wherein everyone in a room cansee and hear the same things. It supports the concept of a virtual room wherein all


users who have entered the room can see and hear each other, interact with the sameapplications, and see the same output from these applications. Another importantsynchronous collaboration model, theaura model, is based on the metaphor ofobjects navigating through a 3-D space. In this model, the 3-D space is virtual andthe navigating objects include both humans and applications. These objects caninteract with each other when they get close enough, much as humans can interactwith each other when they are close enough to see and hear each other. Severalsystems such as DIVE and MASSIVE are based on this model (Benford et al.,1996).

To support asynchronous collaboration, Dourish uses a diverge-merge model inProspero (Dourish, 1998a), wherein collaboration consists of creating alternativeversions of a base object, merging them back into the base object, creating alter-native versions of the merged object, and so on. The time between merges is ameasure of the asynchrony in the collaboration.

To support both synchronous and asynchronous collaboration, we havedeveloped a different model of collaboration, which is higher-level than the asyn-chronous diverge-merge model and lower-level than the synchronous rooms model.Our work has generalized the metaphor of editing supported by various kindsof editors such as text and graphics editors. We first developed a generalizedsingle-user editing model (Dewan and Solomon, 1990), which models single-userinterfaces. According to this model, an application can be considered an editorof a display of data structures. A user interacts with the application by editing thedisplay using text/graphics editing commands. Thus, interaction with an interactiveapplication is similar to interaction with a text or graphics editor. The differenceis that the display is “active”, that is, changes to it can trigger computations in theapplication, which can update the display and send messages to other applications.As a result, the model also includes, for instance, a spreadsheet, which can beconsidered an editor of a matrix that responds to an editing of an entry in the matrixby updating related entries.

We have been extending this model incrementally to support various collabo-ration functions. We first developed a transaction-based collaborative interactionmodel (Dewan and Choudhary, 1992), which allows multiple users to edit differentversions of the application data structures and transmits a user’s changes to otherusers when the changes are committed. We later included in it a coupling modelto support flexible coupling requirements (Dewan and Choudhary, 1995), therebyallowing, for instance, the users to choose whether they share committed or uncom-mitted results. We then integrated the classical access control model (Graham andDenning, 1972) and the US&R undo model (Archer et al., 1984) with our editingand coupling models (Choudhary and Dewan, 1995; Dewan and Shen, 1998b).Similarly, we developed merge and concurrency control modes for collaboration(Munson and Dewan, 1994, 1996).

92 PRASUN DEWAN

In general, identifying an interaction model before designing the user-interfaceoffers the benefits of stepwise refinement, supports uniformity, provides a basis forautomation (as an abstract interaction model may be mapped to multiple concreteuser-interfaces), and allows a user-interface to be based on some familiar metaphor,increasing the chances that it will be easily understood and adopted by users.

2.7. USER INTERFACE

The next step is to implement the concrete user-interface, which we will not expandon here, since it does not have much to do with collaboration, and has more to dowith the user-interface tool and look-and-feel adopted by the system. This step maynot be taken by an infrastructure, which may leave it to the application programmerto work out the user-interface details. For instance, Prospero does not provide auser-interface for merging, while Suite does.

2.8. ARCHITECTURE

The architecture of a software system (Calvary et al., 1997; Dewan, 1998; Rosemanand Greenberg, 1998) determines how the implementation is partitioned intomodules, layers, threads, processes, replicas, and hosts. We refer to the division ofthe system into modules, layers, threads, and processes as thelogical architectureof the system, since the programmer is aware of it when developing the system; andthe partitioning into replicas and hosts as thephysical architectureof the system,as it can be determined dynamically by an external agent.

Like the other steps we have seen so far, how exactly the steps of identifying thelogical and physical architecture are taken are research issues for which there areno standard approaches. For instance, a collaborative system can be implementedas a single, monolithic module; or separated into a semantic module responsible forimplementing the functionality of the application and a sharing module responsiblefor allowing multiple users to access the functionality (Hill et al., 1994). Similarly,a module may execute at a central host, or be replicated on the host of every userinteracting with it.

In general, the architecture(s) of a collaborative system can influence thereusability, efficiency, and complexity of the system. For instance, partitioning acollaborative system into separate functional and sharing modules allows the samefunctional module to be used with different collaborative modules, and vice versa.Similarly, replicating a module gives better response times since a replica executeson the local host of user. On the other hand, the replicated approach often requirescomplex synchronization. An architecture can also determine the semantics ofthe system. For instance, replicating a module will execute the operations of themodule multiple times, once for each user, while centralizing a module will executethem once. If these actions include non-idempotent operations such as file writes,


then the two architectures will yield different semantics. The different ways ofpartitioning an application are related to each other. For instance, the more fine-grained the division of an application into modules, the more flexibility there is inindependently placing different pieces of code on different computers. In particular,if the code executing idempotent and non-idempotent operations is separated intodifferent modules, as in Shasthra (Anupam and Bajaj, 1993), then the former canbe replicated and the latter centralized.

Thus, it is necessary to carefully identify the architecture of a collabora-tive system. Ideally, it would be useful to identify a well-accepted, fine-grained,application-independent “reference architecture” for a collaborative system, likethe “scanner-parser-semantic routines” architecture for a compiler. Such an archi-tecture would allow different research efforts to compare and use each others’components.

There has been some recent work in identifying application-specific andapplication-independent architectures. All shared window systems support anarchitect use in which a collaborative application is divided into a collaboration-transparent window client and a collaboration-aware window server; and most ofthe higher-level systems support one in which it is divided into a collaboration-transparent model and collaboration- aware views (Krasner and Pope, 1988).There has been limited work in dividing collaboration-aware software into separatemodules such as session manager (Ter Hofte, 1998; Roseman and Greenberg,1996), latecomer manager (Chung et al., 1998), coupler (Grundy, 1998), concur-rency controller (Dewan and Sharma, 1999; Grundy, 1998), and access controller(Syri, 1997; Trevor et al., 1994). However, the components identified so far havebeen developed by separate groups and do not necessarily fit together. Moreover,they do not currently offer the full functionality provided by monolithic systemssuch as Suite.

Based on the discussion above, it might seem that there is a one-to-one mappingbetween the functions into which an application’s semantics is decomposed andthe modules into which its implementation is partitioned, since the independentmodules mentioned above correspond to the independent functions identified inSection 2.2. This does not always have to be the case, however: A function maybe implemented by multiple modules. For instance, session management may haveseparate modules for creating the user-interface for a new user and updating it toreflect the current conference state. Conversely, a single module may implementmultiple functions. For instance, an access-control module that can disable arbit-rary user actions can also be used for locking-based concurrency control. Similarly,it is possible to decompose a collaborative system into components that are inde-pendent of functionality. For instance, the latest version of MASSIVE (Greenhalghand Benford, 1995), MASSIVE-3, decomposes a virtual world into layered unitscalled locales that represent different regions in the world and can be independentlyreplicated.

94 PRASUN DEWAN

As is usual in most software development, the first attempt at implementingsome new functionality is a monolithic system. In is only after the nature of thissystem is understood that it is possible to separate the implementation into looselycoupled components. As the works above show, the field is now mature enough tonow think of components, but not mature enough, as say compilation is, to havedeveloped a general reference logical or physical architecture.

2.9. ABSTRACTIONS

Abstractions describe an infrastructure from the point of view of the programmerswho use the infrastructure. They allow programmers to implement the interactionmodels and architectures supported by the infrastructure. As example of a collab-orative abstraction is the shared-window abstraction, which is a logical windowshared by multiple users. The abstraction has been designed to support the roomsinteraction model in shared window systems. Other examples of collaborativeabstractions are a Colab replicated object (Stefik et al., 1987), and the Rendezvousshared object (Hill et al., 1994). In a componentized infrastructure, an abstractionmight be provided as a generic default component that may be extended or replacedby the application.

It is in the design of abstractions that the ease-of-specification requirement ofinfrastructures must be considered, since it is possible to define both low-level andhigh-level abstractions supporting the same functionality. For instance, a generalpurpose distributed programming environment can allow programmers to imple-ment the rooms (and any other) collaboration model but, compared to sharedwindow systems, the abstractions offered by it are low-level.

In theory, abstractions supporting multiuser interaction can be independent ofabstractions supporting single-user interaction. However, in practice, designers ofcollaborative abstractions have extended single-user abstractions, thereby reusingthe design of the non-collaborative aspects of these abstractions. For instance,shared windows are extensions of single-user windows and Colab replicatedobjects are extensions of single-user objects.

Our own work has developed an abstraction, called a shared active variable,to provide high-level support for the editing interaction model. A shared activevariable is an extension of a single-user active variable. A single-user active vari-ables is a program variable that can be modified by a single user and allowsthe program to execute semantic actions in response to user modifications. It isassociated with an interaction variable, which is local buffer edited by the userto change the corresponding active variable. Changes to the interaction variableare communicated to the client when the variable is committed by the user, whichcan update the corresponding active variable and execute the semantic actions. Anactive variable provides high-level support for the generalized single-user editingmodel since programmers are responsible only for specifying the active variablesand high-level user-interface attributes of the corresponding interaction variables.


The infrastructure uses this information to automatically create a user interface forediting these variables.

A shared active variable is an extension of a single-user active variable that canbe edited by multiple users. For each user, a separate interaction variable is created,which can be edited by the user to modify both the associated active variable andthe corresponding interaction variables created for other users. The interaction vari-ables are associated with several collaboration attributes, which determine how thevarious collaboration functions are preformed. For instance, one of these attributesdetermines whether the views of corresponding interaction variables created fordifferent users are coupled.

An abstraction can be incrementally developed as the model is incrementallyextended. For instance, we first developed a shared active variable supporting fixedcoupling and then incrementally extended it to support a flexibly coupled variable,a protected active variable, a recoverable active variable, and a mergeable activevariable.

2.10. INFRASTRUCTURE IMPLEMENTATION

It is in this step that the collaborative abstractions are implemented. The maindecision to be made at this point is whether the implementation of the collaborativeinfrastructure can/should reuse the implementation of an existing infrastructure.This approach is attractive when the abstractions offered by the infrastructure areextensions of existing abstractions since the extended functionality does not haveto be reimplemented.

An extreme form of this approach is to reuse existing implementations withoutmaking any modification to them. This approach is used, for instance, in theimplementation of XTV (Abdel-Wahab and Jeffay, 1994), which uses the imple-mentation of the X server without making any changes to it. The advantage ofthis approach is that it makes the infrastructure implementation independent ofan existing implementation. For instance, the XTV implementation can use anycurrent or future implementation of the X server that supports the X protocol. Adisadvantage is that the new implementation must reconstruct the state managedby the existing implementation, thereby duplicating effort. For instance, in theimplementation of XTV, the window state maintained by the X server must bereconstructed in the new implementation to allow this state to be automaticallycrated for a late comer joining the collaborative session (Chung et al., 1993).Because of this drawback, some projects such as HP’s Shared X (Garfinkel et al.,1994) have modified existing implementations. This was also the approach taken inmultiuser Suite, which was implemented by modifying the existing implementationof single-user Suite.

96 PRASUN DEWAN

2.11. APPLICATION DEVELOPMENT

An application may be developed with or without an infrastructure. If it isdeveloped from scratch, then it must, of course, implement all of the collabo-ration functions. If it is developed using an infrastructure, it is responsible forthree main tasks: First, it must process input events sent by the infrastructure. Forinstance, an XTV client must process X events, and a Suite client must processcallbacks invoked by Suite to inform it of changes to interaction variables. Second,it must override the default values of the parameters offered by the infrastructurethat it wishes to customize. For instance, Suite, by default allows sharing of onlycommitted results. A Suite client that wishes to offer sharing of uncommittedchanges must override the default value of the appropriate coupling parameter.Finally, it must offer its own high-level user-customizable parameters if the para-meters offered by the infrastructure are too low-level. For instance, Suite offersseveral parameters to support a whole spectrum of couplings ranging from nocoupling to very tight, WSYIWIS coupling. An application that expects its usersto use only a few of these points might define a higher-level parameters to choosefrom the smaller set, which could include, for instance, ‘loose’, ‘medium’, and‘tight’ couplings.

If the abstractions are high-level, then the application developer is concernedmainly with the design of the (collaborative aspects of the) application since theimplementation details are taken care of by the infrastructure. The implementationcan become challenging if either the abstractions are low-level or the implement-ation of an existing single-user application must be reused. For instance, in ourimplementation of the FLECSE collaborative debugger, we decided to reuse theimplementation of the complex semantics of an existing single-user debugger. Theimplementation effort, however, was non-trivial since the collaborative debuggerhad to recreate the user interface and much of the internal state (such as the stack)of the single-user debugger.

The “application” developed at this stage may in fact be a prototype whosefunction is to identify scenarios and requirements. In this case, the next step offormal evaluation would not be taken. Instead, the scenarios and requirement stepswould be revisited using the feedback arrows from this step.

2.12. EVALUATION

This phase determines how well the collaborative infrastructure and applicationsmeet their requirements and also how well these requirements reflect actual usersneeds. We might want to know absolute evaluation information such as the exactresponse times of user commands, or relative information such as whether theresponse times of a system are better than those of another one implementing thesame set of commands. Feedback from this phase results in further iterations of therequirements and other phases. A comprehensive evaluation phase requires severalkinds of evaluations.


2.12.1. Inspection

A simple but effective evaluation method is to inspect the collaborative system todetermine if a requirement has been met. For instance, it is possible to determine ifthe requirement of variable-grained locking has been met by simply inspectingthe system design.2 This step cannot be taken, of course, if the requirementshave not been explicitly identified. This approach can be used for some of thespecific requirements regarding the flexibility of the system and also for the ruserequirement. However, it cannot be used for the ease of specification and efficiencyrequirements, which require other evaluation mechanisms. Most research effortsuse this mechanism to some extent. Our work on Suite (Dewan et al., 1994) andRoseman and Greenberg’s work on GroupKit (Roseman and Greenberg, 1996) useit as the main evaluation mechanism.

2.12.2. Simulations

Another evaluation method is to determine if some function of a collaborationsystem can simulate the corresponding function of another existing system. Thismechanism can be used to determine if the former is at least as flexible as thelatter. It can be also used to determine how easy to specify the former is comparedto the latter since the size of the simulation gives an indication of the effort requiredto describe a functionality in the former that is provided directly by the latter. Toillustrate, we used the Suite coupling function to simulate, with a few parametersettings, the coupling functions of shared window and other systems (Dewan andChoudhary, 1995), thereby showing that the former is more flexible than thesesystems without being unduly low-level. We have similarly simulated existingaccess-control policies (Dewan and Shen, 1998a, 1998b), merge policies (Munsonand Dewan, 1994), and concurrency control policies (Munson and Dewan, 1996).To the best of our knowledge, no one else has adopted this approach.

2.12.3. Standard problems

One technique to evaluate the ease of specification of systems is to create completesolutions required to implement standard problems and let the evaluators of asystem judge it based on these solutions. Since it is difficult to agree on thesemantics of complex real problems or to understand and evaluate solutionsto them, these standard problems tend be simple, toy problems such as thedining-philosopher problem used to evaluate concurrent programming languages.However, it is an efficient technique for evaluating a system, since it gives the eval-uator complete information regarding the solution to the problem without requiringthem to use the system. It is particularly useful when a large number of systemshave to be compared with each other.

For applications, this approach implies documenting all the steps taken by aset of collaborators to perform some standard task; whereas for infrastructures, itimplies encoding complete programs implementing some standard applications.

98 PRASUN DEWAN

Figure 2. A Multiuser Outline Editors. Reprinted courtesy of ACM, (c) 1992.

However, currently, there is no set of collaborative tasks or applications that areconsidered to be standard.

In our paper on FLECSE (Dewan and Riedl, 1993), we describe candid-ates for standard collaboration tasks and give the steps taken by a set of threecollaborators to solve the tasks. The inspection scenario given in § 2.3 gives asubset of the steps required to do the tasks. In demonstrates benefits of computersupported inspection such as allowing distributed users to inspect a document. TheDARPA evaluation report identifies an emerging repository (stored currently on theWeb at http://www.antd.nist.gov/∼ icv-ewg/pages/scenarios.html) of collaborationtasks.

Malone et al. (1992) identify several candidates for standard asynchronouscollaborative applications such as gIBIS (Conklin and Begeman, 1988), Coordin-ator (Flores et al., 1988), and Lotus Notes; and describe their experience withimplementing these applications using the Oval infrastructure. Roseman andGreenberg have identified an interesting candidate for synchronous collabora-tive applications – a true “hello world” program that actually greets the “world”rather than a single user (Roseman and Greenberg, 1996), and have given thecomplete code required to implement it using GroupKit. We have identified severalother candidates for several synchronous collaborative applications including asimple multiuser talk application, text editor, outline editor, and spread-sheet, andexplained how they can be implemented using Suite abstractions (Dewan andChoudhary, 1992). Figure 2 illustrates the semantics of the simple multiuser outlineeditor, where the two windows show the display seen by two different users ofthe editor. An outline is a list of sections, where each section has a name field,a contents field, and a list of subsections. Users can edit any of these fields and‘elide’ or condense the display of a section.

Figure 3 gives the complete code of the Suite program that implements theeditor. The outline structure is described by C type declarations, which defineAnOutline as a sequence ofASection records, whereASection has fields forstoring the components of a section. The special comment beginning withdmc tellsSuite that values of typeAnOutline will be made active variables. The first state-


Figure 3. Suite Implementation of Multiuser Outline Editor.

ment in the Load procedure tells Suite to create a new active variable, whose nameis Outline, address is&Outline, and type isAnOutline. The second statementtells Suite to display the variable. Between these two statements, a programmer canput statements overriding the default attributes of the variable. This program usesall the defaults.

As illustrated by the example, complete programs can be an effective way forjudging the properties of a collaborative infrastructure. For instance, this exampleshows how easy it is to implement collaborative textual structure-editors in Suite.A programmer has to simply define a C type describing the structure and create anactive variable of that type. The system takes care of connecting multiple users tothe application, displaying the active variable to the users, allowing them to edit it,coupling their changes, and implementing other collaboration functions.

This approach has also been taken in the Trellis (Stotts and Furuta, 1989) andCOCA (Li and Muntz, 1998) projects, which show how their systems can be usedto specify various control protocols for structured meetings.

It may not be possible to implement all of the standard applications using aparticular infrastructure. For instance, the multiuser card game described in (Hillet al., 1994), which would be a good candidate for a standard application, cannotcurrently be supported by Suite. Similarly, it may not be possible to performall of the standard tasks using a particular collaborative application. The list ofstandard tasks/applications that can be supported by a system is an evaluation ofits flexibility.

None of the evaluation steps described so far involve actual use of a system.As a result, they can be taken incrementally, as each function is researched. Userstudies, which are the next step, require systems that are complete enough to beusable for some collaboration task.

100 PRASUN DEWAN

2.12.4. User studies

The actual experience of the end users with a collaborative application candetermine whether (a) some scenarios desired by users (e.g. asynchronous inspec-tion) were not anticipated, (b) the hypothesized scenarios were actually realisedand found useful, (c) the parameters provided by the application were high-level,and (d) response times of the application were adequate. These data evaluate notonly a particular collaborative application but also indirectly the infrastructuresince, as mentioned in § 2.5, properties (ease-of-specification, efficiency, and flex-ibility) of the infrastructure influence corresponding properties of the applications.For instance, if users of the collaborative inspector find asynchronous annotationsuseful, then this information implies that an infrastructure should support asyn-chronous collaboration. Usage data can be collected by recording user answers toquestions or observing measurable usage data such as time taken to complete a taskand characteristics of the end-product (e.g. number of bugs found in a collaborativeinspection (Dewan and Riedl, 1993).

We have identified at least three stages in the gathering of usage data:

• Self Studies: The developers of an application themselves use the application.• Lab Studies: Users other than the developers use the application to solve

representative problems given to them.• Field Studies: Target users of the application use it to do real work.

The advantage of gathering the data in multiple stages is that feedback from oneset of users can be used to refine the application before it is evaluated by a set of‘less forgiving’ users, that is, users who would find problems with the applicationmore disruptive.

Our research has partially followed this plan. For instance, we have collectedusage data from a self study of a visitor scheduling application (Dewan, 1993c),and a lab study of the problem of inspecting the design of a hospital managementsystem (Mashayekhi et al., 1993). We have not yet progressed to the third stagethough our experience with the first two stages has been positive. Examples ofcollaboration technologies that have been tested in the field include Lotus Notes,a widely deployed technology, which was used by the program committee ofthe CSCW’94 and CSCW’96 conferences to review papers; PoliTeam (Prinz andKolvenbach, 1996), a workflow system used in a distributed German ministry;Babble (Bardner et al., 1999), a chat tool used in software engineering and otherdomains; and GroupLens (Resnick et al., 1994), whose technology for groupfiltering is used in several commercial systems including Amazon.

2.12.5. Programming data

Similarly, the actual experience of the programmers with a collaborative infrastruc-ture can determine whether they found infrastructure abstractions to be high-level.These data can be collected by recording the answers of the programmers to


questions or observing measurable data such as lines of code or time required toimplement applications.

We have identified the following stages in the gathering of programming data:• Self Prototyping: The developers of the infrastructure use it to develop

demonstrational prototype applications.• Self Implementation: The developers of the infrastructure use it to develop lab

applications, that is, applications for which usage data is collected.• Lab Prototyping: Programmers other than the developers use it to develop

prototype applications.• Lab Implementation: Programmers other than the developers use it to develop

lab applications.• Field Implementation: Target programmers use it to develop industrial-

strength real applications.We have followed this plan by doing, for instance, self prototype of a collab-

orative project management application that demonstrates how activities of theresearchers and industrial sponsors participating in the NSF (National ScienceFoundation) Software Engineering Research Center could be coordinated (Dewan,1990), self implementation of a visitor scheduling application (Dewan, 1993c),which was developed by the author and used for a short while to gather usagefeedback; lab prototyping of the FLECSE collaborative debugger (Dewan andRiedl, 1993); and lab implementation of the FLECSE software inspector. We havenot yet progressed to the last step. Field implementation has been carried out forthe GroupKit technology, which is widely used and is now part of a commercialproduct called Team Wave.

2.12.6. Performance data

The performance of collaborative systems is difficult to define and measure. Worstcase times are not always useful since, in our experience, the times for executingthe different algorithms implementing collaboration functions are either constantor linear, and variations in performance arise from difference in the constantsignored in these costs. This is also true in other areas such as database systems andcomputer architectures, which have used two approaches to address this problem.One it to ask users of the system if they find the performance ‘adequate’ and theother is to use benchmarks for evaluating the performance of systems.

Both approaches are appropriate for collaboration systems. For instance, wehave evaluated the performance of our infrastructure and applications by determ-ining whether the response times were adequate in the self and lab studies.Currently, no set of accepted benchmarks exist for collaborative systems, but oncea set of standard scenarios and applications is defined, suitable benchmarks couldbe identified. A benchmark for an application would exercise selected applicationoperations of selected scenarios. A benchmark for an infrastructure would exerciseselected infrastructure operations exercised in selected scenarios involving selectedapplications. We have partly used this approach to measure the costs of method

102 PRASUN DEWAN

invocation and user- interface operations in Suite (Dewan and Choudhary, 1990),the cost of multimedia annotations and teleconferencing in the Suite application,SuiteSound (Riedl et al., 1993), and the cost of migrating collaborative applicationsin XTV (Chung and Dewan, 1996). Similarly, the Clock project has used it tomeasure the performance of a semi-replicated architecture under different cachingconstraints (Graham et al., 1996).

Performance data can be gathered by instrumenting a system (infrastructure orapplication) during the implementation phase, since it is much harder to instrumentlater, during the evaluation phase.

2.12.7. Summary of evaluation phase

The goal of this phase is to evaluate the collaborative systems and the specificrequirements they have been designed to meet. We have identified four top-level requirements, high-level or ease-of-specification, flexibility, efficiency, andreuse. The ease of specification of a collaborative system is evaluated by simu-lations, standard problems, and usage and programming data. Simulations giveinformation regarding the relative ease of specification of the system comparedto other systems. Solutions to standard problems give absolute detailed informa-tion regarding the ease of specification of a collaborative system by showing thecomplete sequence of steps required to solve the problems. Usage and program-ming data give absolute but condensed and potentially subjective information aboutcollaborative applications and infrastructures, respectively.

The flexibility of a system is evaluated by inspections, simulations, and solu-tions to standard problems. Inspections determine the absolute flexibility of acollaborative system by listing the exact flexibility requirements supported by thesystem while simulations give relative information regarding the flexibility of thesystem. Solutions to standard problems also give absolute information regardingthe flexibility of the system by showing which of these problems can be solved bythe system.

The absolute efficiency of a system is measured by performance data. Whetherreuse is supported by a system can be determined by inspecting the design. Finally,the requirements themselves and the hypothetical scenarios from which they havebeen derived are evaluated by usage data identifying which of these scenarios wereactualized in useful collaborations.

3. Conclusions and future work

While few people today question the importance of collaborative systems, there isconsiderable debate as to how these systems should be researched. Based on ourresearch and that of others, we have learned the following main lessons regardingthis issue:• The research can be broken into several phases, which consist of decom-

position of the functionality of collaborative systems into subfunctions that


can be researched independently, adopting general principles regarding thedesign of the function, identifying collaboration scenarios and requirements,adopting an interaction model to meet the requirements, realizing the inter-action model as a concrete user interface, designing a logical and physicalarchitecture, developing infrastructure abstractions, using the abstractions toimplement applications, performing lab studies, field experiments, and simu-lations to evaluate the infrastructure and applications, and finally, derivingdesign and implementation principles applicable to a large number of appli-cations and infrastructures. As in other models with multiple phases, feedbackfrom subsequent phases is used to modify the results from the previous phases.

• To support complex collaboration tasks, collaborative applications and infra-structures should be designed together in an integrated manner by general-izing application-independent design concepts in collaboration infrastructuresand implementing applications using infrastructures.

• Both application-specific and application-independent scenarios should beidentified. These scenarios should be generated by both actual and thoughtexperiments.

• Four important requirements of collaborative systems are efficiency, ease-of-specification, reuse of existing software, and flexibility. These requirementscan conflict with each other and collaborative systems often differ in theway they make tradeoffs among these requirements. These requirementshave different but related meanings for an infrastructure and the applicationsdeveloped using it.

• It is possible to do useful theoretical evaluations (that is, evaluations that donot require any usage experiments) of collaboration systems by inspectingtheir design, simulating existing systems, ad solving standard problems.

• Usage experiments should involve both end-users and programmers. Exper-iments with end-users should consist of self studies, lab studies, and fieldstudies; while experiments with programmers should consist of self proto-typing, self implementation, lab prototyping, lab implementation, and fieldimplementation.

Most of the phases of the research plan are optional, as illustrated by Table I, whichshows (to the best of our knowledge) the extent which some influential projectshave followed this plan. The projects include Suite (Dewan and Choudhary, 1992),MASSIVE (Greenhalgh and Benford, 1995), GroupKit (Roseman and Greenberg,1996), XTV (Abdel-Wahab and Feit, 1991), Oval (Malone et al., 1992), Prospero(Dourish, 1998a), Grove (Ellis et al., 1991), DistView (Prakash and Shim, 1994),and PREP (Neuwirth et al., 1994). In this table, lacking a logical architectureimplies that the project does not require any logical partitioning of the application.All systems have some physical architecture for dynamically starting software ona user’s workstation to interact with other users. Each of the projects is consideredto have done the inspection evaluation since each of these systems, we believe, hassome requirements that are met directly by its design. The infrastructures without a

104 PRASUN DEWAN

Table I. Extent of research plan followed by some representative projects

Suite MASSIVE Group XTV Oval Prospero GROVE Dist PREP

Kit View

Func. Decomp. Yes No Yes No No No No No Yes

Principles Yes No Yes No No Yes No No No

Scenarios Yes Yes Yes Yes Yes Yes Yes Yes Yes

Requirements Yes Yes Yes Yes Yes Yes Yes Yes Yes

Interaction Model Yes Yes Yes Yes No Yes Yes Yes Yes

User Interface Yes Yes Yes Yes No No Yes Yes Yes

Logical Architecture Yes Yes Yes No No No Yes Yes No

Physical Architecture Yes Yes Yes Yes Yes Yes Yes Yes Yes

Abstractions Yes Yes Yes Yes Yes Yes No Yes No

Infra. Impl. Yes Yes Yes Yes Yes Yes NA Yes NA

Applications Yes Yes Yes Yes Yes Yes Yes Yes Yes

Inspection Yes Yes Yes Yes Yes Yes Yes Yes Yes

Simulation Yes No No No No No No No No

Standard Problems Yes No Yes No Yes Yes No No No

Self Studies Yes Yes Yes Yes Yes Yes Yes Yes Yes

Lab Studies Yes Yes Yes Yes No No Yes Yes Yes

Field Studies No Yes Yes Yes No No Yes Yes Yes

user-interface offer low-level abstractions that do not come with any user-interface.The reason for the other entries in the table are presented in the various subsectionsof Section 3.

Some of these research principles such as integrated application-infrastructuredevelopment have been used in other research, as shown by the table – the contri-bution of this paper is the explicit articulation, defense, and most important,illustration of these principles.

This research plan is descriptive in that it explicitly shows the relationshipbetween the various research projects in this area. Thus, it can be used to under-stand the scope of the research in this area. It is also prescriptive in that it presents amodel for future research in this area. In particular, it allows new research projectsto focus in-depth on certain components of the plan without carrying out the wholecomplex plan of designing and evaluating a collaborative system. For instance,it has allowed the in-depth research of coupling, access control, undo, merging,and application development to be carried out by different sets of researchers inour group. It also shows that the end results of successful projects in collaborativesystems are not just software systems but also architectures, abstractions, inter-action models, scenarios, and design principles, which can be expected to havesuccessfully longer lifetimes.


This paper also helps address an important interdisciplinary debate in this area.There are currently two opposing viewpoints shared by researchers of collaborativesystem. One is that research of collaborative systems (like the traditional researchof operating systems, compilers, programming language and other traditional soft-ware systems) should be carried out exclusively by computer scientists. The otheris that any collaboration effort must involve researchers from other disciplines suchas psychology, sociology, and anthropology.

The plan explicitly shows where researchers from areas other than computerscience can most contribute to collaboration research. One of the most fundamentalphases in the plan is identification of scenarios, and these researchers can bringvaluable experience regarding how people collaborate in real life. It is for thisreason that the research plan of several groups includes the use of ethnographers,ethnomethodologists, and/or psychologists in the requirement phase (Bentley etal., 1992a, b; Bowers and Martin, 1999; Kaplan et al., 1994; Pycock and Bowers,1996). Similarly, they can play a crucial role in gathering and analyzing usagedata (Bowers et al., 1996a, b; White, 1994). Projects in more traditional softwaresystems do not have to be interdisciplinary since their requirements are relativelywell established and they do not fundamentally change the way people do theirwork.

However, this does not imply all research efforts in this area must be inter-disciplinary to an equal extent, for various reasons. It is possible to do usefultheoretical evaluations (that is, evaluations that do not require any usage exper-iments) of collaboration systems by inspecting their design, simulating existingsystems, and solving standard problems. Moreover, projects addressing perfor-mance (Chung and Dewan, 1996; Graham et al., 1996), automation (Dewan andChoudhary, 1992), reuse (Abdel-Wahab and Jeffay, 1994; Chung et al., 1998) andflexibility (Dewan and Shen, 1998a) that consider well-accepted designs or designsevaluated by others can be carried out entirely by computer scientists. Moreover, ifthe rapid prototyping approach is used, then the importance of identifying realisticscenarios is less since the cost of reacting to late user feedback during actual usageis low. Furthermore, as our project has demonstrated, it is possible for computerscientists to also carry out at least some of the usage studies – in particular selfand lab studies (Dewan and Riedl, 1993). A corollary of this discussion is that it ispossible to do meaningful evaluations of a system that do not involve user studies.

Our work is only a first-cut effort at a comprehensive research plan, and itsuggests several future directions for research. It is important to identify generalsolutions to the high-level phases of the plan so that research projects done in thelower phases can complement each other and be compared. In particular, if everyresearch project in collaborative systems used its own functional decompositionand scenarios, then there would be very little chance of general principles emergingfrom this area. It is the high-level phases, however, that have been largely ignoredin this area, and it is crucial to focus more attention on them.

106 PRASUN DEWAN

It is also important to actually carry out a set of research projects that togetherfollow the model. Our own work has followed all of the steps except the fieldstudies and it is important to do them to validate the plan.

In summary, a general research plan for collaborative systems can help identifythe research agenda for this area. This paper has taken an initial step in thisdirection.

Acknowledgments

Several researchers participated in the research outlined in this paper and contrib-uted implicitly or explicitly in the formulation of this research plan. Discussionswith John Riedl contributed to an initial version of the overall structure givenin Figure 1. The inclusion in the plan of explicit requirements, standard appli-cations, and simulations was motivated by the work of Rajiv Choudhary, JonMunson, and HongHai Shen on undo, access control and merging, respectively,and lab implementations/studies was motivated by Vahid Mashayekhi’s work onthe FLECSE software inspection application. The research was supported in partby National Science Foundation grants IRI-9408708, IRI-9508514, IRE-9627619,IIS-9977362, and CDA-9624662; and by DARPA/ONR grant N 66001-96-C-8507.The insightful comments of the referees significantly improved the paper.

Notes

1. In fact, this work was prompted by the reviews of one of our research proposals, which wantedan explicit description of the research methodology we were using to evaluate our work.

2. Assuming that the design has been implemented and tested.

References

Abdel-Wahab, H.M. and M.A. Feit (1991): Proceedings of IEE Conference on CommunicationsSoftware: Communications for Distributed Applications & Systems.

Abdel-Wahab, Hussein and Kevin Jeffay (1994): Issues, Problems and Solutions in Sharing X Clientson Multiple Displays.Internetworking: Research and Experience, vol. 5, pp. 1–15.

Andriole, Stephen J. (1994): Fast, Cheap Requirements: Prototype, or Else!IEEE Software, pp. 85–87.

Anupam, V. and C. Bajaj (1993): Collaborative Multimedia Scientific Design in SHASTRA.Proceedings of ACM Conference on Multimedia, pp. 447–456.

Archer, James E., Richard Conway and Fred B. Schneider (1984): User Recovery and Reversal inInteractive Systems.ACM Transactions on Programming Language and Systems, vol. 6, no. 1,pp. 1–19.

Bardner, Erin, Wendy A. Kelogg and Thomas Erickson (1999): The Adoption and Use of ‘BABBLE’:A Field Study of Chat in the Workplace.Proceedings of European Conference on ComputerSupported Cooperative Work, pp. 139–158.

Beck, E. and V. Belloti (1993): Informed Opportunism as Strategy.Proceedings of ECSCW’93.


Benford, Steve, Chris Brown, Gail Reynard and Chris Greenhalgh (1996): Shared Spaces: Transpor-tation, Artificiality, and Spatiality.Proceedings of the ACM Conference on Computer SupportedCooperative Work, pp. 77–86.

Benford, S.D., C.M. Greenhalgh, D.N. Snowdown and A.N. Bullock (1997): Staging a PoetryPerformance in a Collaborative Virtual Environment.Proceedings of European Conference onComputer Supported Cooperative Work.

Benford, Steven et al. (1999): Broadcasting On-Line Social Interaction as Inhabited Television.Proceeding of European Conference on Computer Supported Cooperative Work, pp. 179–198.

Bentley, Richard, Tom Rodden, Peter Sawyer and Ian Sommerville (1992a): An Architecture forTailoring Cooperative Multi-User Displays.Proceedings of the ACM Conference on ComputerSupported Cooperative Work, pp. 203–209.

Bentley, R., J. Hughes, D. Randall, T. Rodden, P. Sawyer, D. Shapiro and I. Sommerville (1992b):Ethnographically-Informed Design for Air Traffic Control.Proceedings of Computer SupportedCooperative Work.

Bowers, J., J. Pycock and J. O’Brien (1996a): Practically Accomplishing Immersion.Proceedings ofComputer Supported Cooperative Work, pp. 380–389.

Bowers, J., J. O’Brien and J. Pycock (1996b): Talk and Embodiment in Collaborative VirtualEnvironments.Proceedings of Computer Human Interaction.

Bowers, J. and D. Martin (1999): Informing Collaborative Information Visualization through anEthnography of Ambulance Control.Proceedings of European Computer Supported CooperativeWork.

Brothers, L., V. Sembugamoorthy and M. Muller (1990): ICICLE: Groupware for Code Inspection.Proceedings of the ACM Conference on Computer Supported Cooperative Work, pp. 169–181.

Calvary, Gaelle, Joelle Coutaz and Laurence Nigay (1997): From Single-User Architectural Designto PAC*: a Generic Software Architecture Model for CSCW.Proceedings CHI’97, pp. 242–249.

Choudhary, Rajiv and Prasun Dewan (1995): A General Multi-User Undo/Redo Model.Proceedingsof European Conference on Computer Supported Work, pp. 231–246.

Chung, Goopeel, Kevin Jeffay and Hussein Abdel-Wahab (1993): Accommodating Latecomers inShared Window Systems.IEEE Computer, vol. 26, no. 1, pp. 72–73.

Chung, Goopeel and Prasun Dewan (1996): A Mechanism for Supporting Client Migration in aShared Window System.Proceedings of the Ninth Conference on User Interface Software andTechnology, pp. 11–20.

Chung, Goopeel, Prasun Dewan and Sadagopan Rajaram (1998): Generic and Composable Late-comer Accommodation Service for Centralized Shared Systems. In Chatty and Dewan (eds):Proceedings of IFIP Conference on Engineering for Human Computer Interaction. Dordrecht:Kluwer Academic Publishers, pp. 129–148.

Clauer, C.R., J.D. Kelley, T.J. Rosenberg, C.E. Rasmussen, P. Stauning, E. Friis-Christensen, R.J.Niciejwski, S.B. Mende, Y. Zambre, T.E. Weymouth, A. Prakash, G.M. Olsom, S.E. McDaniel,T.A. Finholt and D.E. Atkins (1994): A New Project to Support Scientific CollaborationElectronically.EOS Transactions on Americal Geophysical Union, vol. 75.

Conklin, J. and M.L. Begeman (1988): gIBIS: A Hypertext Tool for Exploratory Policy Discussion.Proceedings of ACM Conference on Computer Supported Cooperative Work, pp. 140–152.

Cugini, John, Laurie Damianos, Lynette Hirschman, Robyn Kozierok, Jeff Kurtz, SharonLaskowski and Jean Scholtz (1996): Methodology for Evaluation of Collaboration Systems.http://www.aantd.nist.gov/∼icv-ewg/docu-ments/meth_index.html.

Curtis, Pavel (1992): Muddings: Social Phenomena in Text-Based Virtual Reality. CSL-92-4, XeroxPalo Alto Research Center.

Dewan, Prasun (1990):A Guide to Suite, Technical Report SERC-TR-60-P, Software EngineeringResearch Center, Purdue University.

108 PRASUN DEWAN

Dewan, Prasun (1992): Principles of Designing Multi-User User Interface Development Environ-ments.Proceedings of the IFIP TC2/WG 2.7 Working Conference on Engineering for Human-Computer Interaction, Ellivuori, Finland, Amsterdam: North Holland, pp. 35–50.

Dewan, Prasun (1993a): Tools for Implementing Multiuser User Interfaces.Trends in Software: Issueon User Interface Software, Wiley, vol. 1, pp. 149–172.

Dewan, Prasun (1993b): A Survey of Applications of CSCW Including Some in Educational Settings.Proceedings of ED-MEDIA’93, pp. 147–152.

Dewan, Prasun (1993c): Designing and Implementing Multi-User Applications: A Case Study.Software-Practice and Experiencevol. 23, no. 1, pp. 75–94.

Dewan, Prasun (1998): Architectures for Collaborative Applications.Trends in Software: ComputerSupported Co-operative Work, Wiley, vol. 1, pp. 165–194.

Dewan, Prasun (2000): Techniques for Evaluating Collaborative Toolkits.WET-ICE workshop onEvaluating Collaborative Enterprises.

Dewan, Prasun and Marvin Solomon (1990): An Approach to Support Automatic Generation of UserInterface.ACM Transactions on Programming Language and Systems, vol. 12, no. 4, pp. 566–609.

Dewan, Prasun and Rajiv Choudhary (1990): Experience with the Suite Distributed Object Model.Proceedings of IEEE Workshop on Experimental Distributed Systems, pp. 57–63.

Dewan, Prasun and Rajiv Choudhary (1992): A High-Level and Flexible Framework for Imple-menting Multiuser User Interfaces.ACM Transactions on Information Systems, vol. 10, no. 4,pp. 345–380.

Dewan, Prasun and John Riedl (1993): Toward Computer-Supported Concurrent SoftwareEngineering.IEEE Computer, vol. 26, no. 1, pp. 17–27.

Dewan, Prasun and Rajiv Choudhary (1995): Coupling the User Interfaces of a Multiuser Program.ACM Transactions on Computer Human Interaction, vol. 2, no. 1, pp. 1–39.

Dewan, Prasun and HongHai Shen (1998a): Flexible Meta Access-Control for CollaborativeApplications. Proceedings of ACM Conference on Computer Supported Cooperative Work,pp. 59–68.

Dewan, Prasun and HongHai Shen (1998b): Access Control for Multiuser Interfaces.ACM Transac-tions on Computer Human Interactionvol. 5, no. 1, pp. 34–62.

Dewan, Prasun and Anshu Sharma (1999): An Experiment in Interoperating Heterogeneous Collabo-rative Systems.Proceedings of European Conference on Computer Supported Cooperative Work.Dordrecht: Kluwer Academic Publishers, pp. 371–390.

Dewan, Prasun, Rajiv Choudhary and HongHai Shen (1994): An Editing-based Characterization ofthe Design Space of Collaborative Applications.Journal of Organizational Computing, vol. 4,no. 3, pp. 219–240.

Dourish, Paul (1995): Developing a Reflective Model of Collaborative Systems.ACM Transactionson Computer Human Interaction, vol. 2, no. 1, pp. 40–63

Dourish, Paul (1998a): Using Metalevel Techniques in a Flexible Toolkit for CSCW Applications.ACM Transactions on Computer Human Interaction, vol. 5, no. 2.

Dourish, Paul (1998b): Software Infrastructures.Trends in Software: Computer Supported Co-operative Work, Wiley, vol. 7, pp. 195–217.

Ellis, Clarence: (1994): A Conceptual Model of Groupware.Proceedings of the ACM Conference onComputer Supported Cooperative Work, pp. 79–88.

Ellis, Clarence, Simon J. Gibbs and Gail L. Rein (1991): Groupware: Some Issues and Experiences.CACMvol. 34, no. 1, pp. 38–58.

Ensor, J.R., S.R. Ahuja, D.N. Horn and S.E. Lucco (1988): The Rapport Multimedia Conferen-cing System: A Software overview.Proceeding of the 2nd IEEE Conference on ComputerWorkstations, pp. 52–58.


Flores, F., M. Graves, B. Hartfield and T. Winograd (1988): Computer Systems and the Designof Organizational Interaction.ACM Transactions on Office Information Systems, vol. 6, no. 2,pp. 153–172.

Garfinkel, Daniel, Bruce Welti and Thomas Yip (1994): HP Shared X: A Tool for Real-TimeCollaboration.Hewlett-Packard Journal, vol. 45, no. 4, pp. 23–24.

Gintell, John, John E. Arnold, Michael Houde, Jacek Kruszelnick, Roland McKenney and GerardMemmi (1993): Scrutiny: A Collaborative Inspection and Review System. Tech. ReportRAD/USARL/93010, Bull Research and Advanced Development.

Graham, G.S. and P.J. Denning (1972): Protection – principles and practice.Proceedings of SpringJt. Computer Conference, vol. 40, pp. 417–442.

Graham, T.C. Nicholas, Tore Urnes and Roy Nejabi (1996): Efficient Distributed Implementationof Semi-Replicated Synchronous Groupware.Proceedings of the Ninth Conference on UserInterface Software and Technology, pp. 1–10.

Greenberg, Saul and David Marwood (1994): Real-Time Groupware as a Distributed System:Concurrency Control and its Effect on the Interface.Proceedings of CSCW’94.

Greenberg, Saul, Mark Roseman, Dave Webster and Ralph Bohnet (1992): Issues and ExperiencesDesigning and Implementing Two Group Drawing Tools.Proceedings of 25th Annual HawaiiInternational Conference on System Sciences, vol. 4, pp. 139–150.

Greenhalgh, Christ and Steven Benford (1995): MASSIVE: A Collaborative Virtual Environment forTeleconferencing.ACM Transactions on Computer-Human Interaction, pp. 239–261.

Grundy, J. (1998): Engineering Component-based, User-Configurable Collaborative Editing Systemssystems. In Chatty and Dewan (eds):Proceeding of IFIP Conference on Engineering for HumanComputer Interaction, Dordrecht: Kluwer Academic Publishers, pp. 111–128.

Gutwin, Carl and Saul Greenberg (1999): The Effects of Workspace Awareness Support on the Usab-ility of Real-Time Distributed Groupware.ACM Transactions on Computer Human Interactionvol. 6, no. 3, pp. 243–281.

Harper, R., J. Hughes and D. Shapiro (1991): Harmonious Working and CSCW: ComputerTechnology and Air Traffic Control. InStudies in Computer Supported Cooperative Work,Amsterdam: North Holland, pp. 65–80.

Heath, C. and P. Luff (1991): Cooperative Activity and Technological Design: Task Coordinationin Underground Control Rooms.Proceeding of European Conference on Computer SupportedCooperative Work.

Hill, Ralph, Tom Brinck, Steven Rohall, John Patterson and Wayne Wilner (1994): The Rendez-vous Architecture and Language for Constructing Multiuser Applications.ACM Transactions onComputer Human Interaction, vol. 1, no. 2, pp. 81–125.

Hofte, Henri ter (1998):Working Apart Together: Foundations for Component Groupware. Enschede:Telematica Institute.

Hollan, Jim and Scott Stornetta (1992): Beyond Being There.ACM CHI’92 Proceedings, pp. 119–126.

Johnson, Philip (1992): Supporting Exploratory CSCW with the EGRET Framework.Proceeding ofthe ACM conference on Computer Supported Cooperative Work, pp. 298–305.

Kaplan, Simon M., William J. Tolone, Douglas P. Bogia and Celsina Bignoli (1992): Flexible, ActiveSupport for Collaborative Work with ConversationBuilder.Proceedings of CSCW’92.

Kaplan, Simon, Susan Leigh Star, William J. Tolone and Celsina Bignoli (1994): Groundingthe Metaphor of Space in CSCW: Meta-Structures and Boundary.Schaerding Conference onInterdisciplinary Approaches to System Analysis and Design.

Karsenty, Alain, Christophe Tronche and Michel Beaudouin-Lafon (1993): GroupDesign: SharedEditing in a Heterogeneous Environment.Usenix Computing Systems, vol. 6, no. 2, pp. 167–195.

Krasner, Glenn E. and Stephen T. Pope (1988): A Cookbook for Using the Model-View-ControllerUser Interface Paradigm in Smalltalk-80.Journal of Object-Oriented Programmingvol. 1, no. 3,pp. 26–49.

110 PRASUN DEWAN

Lantz, Keith A., Peter P. Tanner, Carl Binding, Kuan-Tsae Huang and Andrew Dwelly (1987):Reference Models, Window Systems, and Concurrency.Computer Graphics, vol. 21, no. 2, pp.87–97.

Li, Du and Richard Muntz (1998): COCA: Collaborative Objects Coordination Architecture.Proceeding of Computer Supported Cooperative Work, pp. 179–188.

Malone, Thomas W., Christopher Fry and Kum-Yew Lai (1992): Experiments with Oval: A RadicallyTailorable Tool for Cooperative Work.Proceedings of CSCW’92, pp. 289–297.

Manohar, Nelson R. and Atul Prakash (1995): The Session Capture and Replay Paradigm for Asyn-chronous Collaboration.Proceedings of European Conference on Computer Supported Work,pp. 149–164.

Mashayekhi, Vahid, Janet Drake, Wei-Tek Tsai and John Riedl (1993): Distributed CollaborativeSoftware Inspection.IEEE Softwarevol. 10, no. 5, pp. 66-75.

Munson, Jon and Prasun Dewan (1994): A Flexible Object Merging Framework.Proceedings of theACM Conference on Computer Supported Cooperative Work, pp. 231–242.

Munson, Jon and Prasun Dewan (1996): A Concurrency Control Framework for CollaborativeSystems.Proceedings of the ACM Conference on Computer Supported Cooperative Work,pp. 278–287.

Neuwirth, Christine M., Ravinder Chandhok, David S. Kaufer, Paul Erion, James H. Morris andDale Miller (1992): Flexible Diff-ing in a Collaborative Writing System.Proceedings of ACMConference on Computer Supported Cooperative Work, pp. 147–154.

Neuwirth, Christine, David Kaufer, Ravinder Chandhok and James Morris (1994): Computer Supportfor Distributed Collaborative Writings: Defining Parameters of Interaction.Proceedings of ACMConference on Computer Supported Cooperative Work, pp. 145–152.

Olson, Gary M., Lola J. McGuffin, Eiji Kuwana and Judith S. Olson (1993): Designing Softwarefor a Group’s Needs: A Functional Analysis of Synchronous Groupware.Trends in Software:Special Issue on User Interface Software, Wiley, pp. 129–148.

Prakash, Atul and Michael J. Knister (1994): A Framework for Undoing Actions in CollaborativeSystems.ACM Transactions on Computer-Human Interaction, pp. 295–330.

Prakash, Atul and Hyong Sop Shim (1994): DistView: Support for Building Efficient CollaborativeApplications using Replicated Active Objects.Proceedings of the ACM Conference on ComputerSupported Cooperative Work, pp. 153–162.

Prinz, Wolfgang and Sabine Kolvenbach (1996): Support for Workflows in a Ministerial Environ-ment.Proceedings of Computer Supported Cooperative Work, pp. 199–208.

Pycock, J. and J. Bowers (1996): Getting Others to Get it Right.Proceedings of ACM ComputerSupported Cooperative Work.

Resnick, Paul, Neophytos Iacovou, Mitesh Suchak, Peter Bergstrom and John Riedl (1994): Group-Lens: An Open Architecture for Collaborative Filtering of Netnews.Proceedings of ACMConference on Computer Supported Cooperative Work, pp. 175–186.

Riedl, J., V. Mashayekhi, J. Schnepf, M. Claypool and D. Frankowski (1993): SuiteSound: ASystem for Distributed Collaborative Multimedia.IEEE Transactions on Knowledge and DataEngineering, vol. 5, no. 4, pp. 600–609.

Roseman, Mark and Saul Greenberg (1996): Building Real-Time Groupware with GroupKit, AGroupware Toolkit.ACM Transactions on Computer-Human Interaction, vol. 3, no. 1, pp.66–106.

Roseman, Mark and Saul Greenberg (1998): Groupware Tooklits for Synchronous Work.Trends inSoftware: Collaborative System, Wiley, vol. 7, pp. 135–164.

Saltzer, J.H. (1974): Protection and the control of information sharing in multics.Communicationsof the ACM, vol. 17, no. 7, pp. 388–402.

Schuckmann, Christian, Lutz Kirchner, Jan Schummer and Jorg M. Haake (1996): DesigningObject-Oriented Synchronous Groupware with COAST.Proceedings of the ACM Conferenceon Computer Supported Cooperative Work, pp. 21–29.


Sikkel, Klaas (1997): A Group-based Authorization Model for Cooperative Systems.Proceedings ofEuropean Conference on Computer Supported Cooperative Work, pp. 345–361.

Stefik, Mark, Gregg Foster, Daniel G. Bobrow, Kenneth Kahn, Stan Lanning and Lucy Suchman(1987): Beyond the Chalkboard: Computer Support for Collaboration and Problem Solving inMeetings.CACM, vol. 30, no. 1, pp. 32–47.

Stotts, P.D. and R. Furuta (1989): Petri Net Based Hypertext: Document Structure with BrowsingSemantics.ACM Transactions on Information Systems, vol. 7, no. 1, pp. 3–29.

Stotts, P.D. and R. Furuta (1994): Trellis: Process Models as Multi-reader Collaborative Hyperdocu-ments.Processing of the 9th Annual Software Process Workshop.

Suchman, L. (1987):Plans and Situated Actions: The Problem of Human-Machine Communication.New York, NY: Cambridge University Press.

Syri, Anja (1997): Tailoring Cooperation Support through Mediators.Processing of EuropeanComputer Supported Cooperative Work, pp. 157–172.

Trevor, Jonathon, Tom Rodden and John Mariani (1994): The Use of Adapters to Support Cooper-ative Sharing.Proceedings of the ACM Conference on Computer Supported Cooperative Work,pp. 219–230.

White, Tedd N. (1994): Applying Scenarios and Claims Analysis as Task Analysis Method.Schaerding Conference on Interdisciplinary Approaches to System Analysis and Design.

Documents

An Integrated Approach to Designing and Evaluating Collaborative Applications and Infrastructures