A Laboratory Method for Studying Activity AwarenessGregorio Convertino (1), Dennis C. Neale (2), Laurian Hobby (2),

John M. Carroll (1) and Mary Beth Rosson (1)

(1) School of Information Science and Technology,The Pennsylvania State University

University Park, PA 16802{gconvertino, jcarroll, mrosson}@ist.psu.edu

+1 814 863 2476

(2) Center for Human-Computer Interaction,Department of Computer Science

Virginia Tech, Blacksburg, VA 24061{dneale, lhobby}@vt.edu

+1 540 231 7542

ABSTRACTMany failures in long-term collaboration occur because of alack of activity awareness. Activity awareness is a broadconcept that involves awareness of synchronous andasynchronous interactions over extended time periods. Wedescribe a procedure to evaluate activity awareness andcollaborative activities in a controlled setting. The activitiesused are modeled on real-world collaborations documentedearlier in a field study. We developed an experimentalmethod to study these activity awareness problems in thelaboratory. Participants worked on a simulated long-termproject in the laboratory over multiple experimentalsessions with a confederate, who partially scripted activitiesand probes. We present evidence showing that this methodrepresents a valid model of real collaboration, based onparticipants' active engagement, lively negotiation, andawareness difficulties. We found that having the ability todefine, reproduce, and systematically manipulatecollaborative situations allowed us to assess the effect ofrealistic conditions on activity awareness in remotecollaboration.

Author KeywordsAwareness, activity awareness, coordination, CSCW

ACM Classification KeywordsH5.m. Information interfaces and presentation (e.g., HCI):Miscellaneous.

INTRODUCTIONEvaluation is crucial to both designers and researchers: itallows the former to verify that a design proposal meetsusers’ requirements, and gives the latter an opportunity toformulate and refine their theories and models of human-

computer interaction (HCI). For user-centered design ingeneral, design and evaluation methods have evolved inparallel. However, in computer-supported cooperative work(CSCW), considerable effort has been put into the design ofnew systems, with much less attention to the systematicevaluation of CSCW systems [26]. This deficiency inmethodology may be due to the intrinsic complexity thatcharacterizes processes, products, and contexts ofcollaboration [13]. Additionally, a fundamental limitationof existing approaches to evaluate distributed CSCWsystems is that they tend to be method-driven rather thantheory-driven. Consequently, they do not sufficientlyinform researchers about how specific methods map tospecific constructs in CSCW, and what critical questionsshould be answered [21].

Researchers in HCI have developed precise methods forevaluating interactions between an individual and acomputer — for example: controlled experiments, time anderror studies of human performance, and mathematicalmodels of human behavior (e.g. Fitts’ Law). But forCSCW researchers, there remain many open questionsconcerning how if at all collaboration can be studied withcost-effective methods and in controlled settings withoutcompromising the ecological validity of the studiedphenomena.

In this paper, we describe a method developed to evaluatecollaborative activities in a controlled setting. The activitiesinvoked are modeled on real world collaborationsdocumented earlier in a field study. More specifically, wedeveloped the laboratory methods to study a pervasivecollaborative phenomenon that impacts the success ofCSCW systems - activity awareness [4]. Activity awarenesssubsumes the prior notions of awareness in collaboration(e.g., workspace awareness or awareness of asynchronouscollaboration) and refers to the understanding thatcollaborators have of prior, current, and future activity.Although we designed and implemented a method to studyactivity awareness, this approach can be used to study otheraspects of collaboration that are heavily dependent oncontext and that arise during extended collaborationactivities.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.

NordiCHI '04, October 23-27, 2004 Tampere, Finland Copyright 2004 ACM 1-58113-857-1/04/10... $5.00

313

Typically, the rich interactions of collaborative work areexamined through field studies and ethnographic methods.On one hand, these observational methods are essential forunderstanding authentic practices, but on the other theydemand large investments of resources and time.Additionally, field methods do not lend themselves to directmanipulation or control of the studied phenomena (a fieldstudy may allow for process or system changes to bestudied, but it is often indirect, uncontrolled, and slow toproduce results). To investigate the relationships betweenspecific contextual events, such as collaborativebreakdowns and their effect on collaborative phenomena(awareness, coordination, common ground, and planning),we need research methods that allow systematicinvestigation of such relationships within semi-naturalisticcontrolled settings.

Researchers emphasize that CSCW evaluation should occurin the context of actual use [31, 25]. Unfortunately,fieldwork evaluation does not mesh well with fast pacedsystem development lifecycles. The procedures adopted inthe field are often opportunistic and based on informalevaluation sessions [31]. As a result, CSCW systems havefailed to a much greater degree than single-user systemsdue to inadequate usability feedback, and more importantly,feedback concerning the proposed functionality.

One approach to more systematic CSCW evaluation is tosimulate real usage conditions: evaluate functionality andusability in semi-controlled conditions designed to mirrorreal-world contexts. Given the difficulty of reproducingsuch a realistic setting, researchers in CSCW have recentlydeveloped discount evaluation methods extending themethods developed for individual-based applications tocollaborative systems (basic inspection [29]; cognitivewalkthrough [23]; heuristic evaluation [2]). Specifically,some authors [14, 2] have advocated the use of heuristicevaluation methods customized for groupware systems.Instead of studying real work situations, groupware systemscan be evaluated by experts with reference to a set ofCSCW usability principles – usage issues influencing themechanics of collaboration [2]. Others have proposed newmodeling techniques, such as Collaboration UsabilityAnalysis [24] (a task analysis technique for studyingteamwork), to provide evaluators with new analysisschemes that are appropriate for groupware usabilityevaluation.

Discount evaluation methods are an importantmethodological contribution because they promote savingsin costs and allow evaluation to take place during earlydevelopment, when there is no operational prototype forusers to test in real work settings [24]. However, they areunlikely to be sufficient to support the full evaluation“lifecycle” of CSCW systems. For example, Cockton andWoolrych have discussed several methodological problemswith discount usability evaluation methods [6]. Theysuggest that evaluation methods should be judged notsimply for their benefits (e.g., rapid feedback), but also for

their cost and risk of error (e.g., poor ecological validity).Continuing this general argument, we propose thatcontrolled laboratory methods should be judged withrespect to both their benefits (control and precision) andrisks (poor ecological validity). More specifically, we arguethat by modeling laboratory tasks on field observations andsimulating them in controlled settings, we can obtain thebenefits of a laboratory study, while at the same timereduce our risk of missing important properties relevant toreal system usage.

To address CSCW evaluation in all stages of development,we present a laboratory method that was conceived as partof a larger research paradigm to study collaborativeactivities through both field and laboratory studies.Although the laboratory methods are grounded in the resultsof field research, ultimately we propose a researchparadigm that is bidirectional: laboratory studies should beable to inform field studies and vice versa. We believe thatdoing research on collaboration by interleaving lab andfield studies will increase the heuristic potential of bothmethodologies, offset the drawbacks of individual methods[19, 3], and reveal new important aspects of the studiedphenomena – in this case activity awareness. In this paper,we describe our first instantiation of the laboratory methodand show that it is a valid CSCW experimental approach.

ACTIVITY AWARENESSThe CSCW problem domain we used to instantiate andvalidate our laboratory methods was the study of the high-level and complex collaborative phenomenon of activityawareness [4].

Adopting the notion of activity to study awareness incollaborative work requires a conceptual shift from aknowledge-centered to an activity-centered perspective.Several studies that have investigated awareness inindividuals and teams have focused on knowledgerepresentations that underlie awareness (e.g., theorizationson situation awareness by Endsley [9] and Salas [27]).These studies have focuses on individual and shared mentalmodels as precursor products of team’s awareness. Weargue that the notion of shared mental models attributesmore emphasis to convergence than divergence amongteam members [20] and characterizes teamwork through itsproducts (knowledge) rather than group process. Thisreduces the explanatory power that this theoretic approachhas on the complex and dynamic phenomenon ofawareness. In fact, awareness is not only dependent onshared knowledge and common ground, but is also directlyrelated to members’ coping with divergent views, resolvingconflicts and uncertainty, adapting to changes over time,and understanding how current activities fit with prioractivities, future plans, members’ roles, norms in the group,and in the larger socio-organizational context.

Understanding, defining, and operationalizing the manyroles of awareness in collaboration is a key problem for thesuccess of CSCW systems [16]. Much of the research in

314

this area has focused on social (who is present) and action(what is s/he doing) awareness. However, field observationshave shown that even when social and action awareness aresupported, many disruptive awareness breakdowns stilloccur. Most of these problems can be attributed to the lackof activity awareness [4]. Activity awareness refers topeople’s ability to get and maintain “the big picture” aboutthe ongoing collaboration while they are working togetheron long-term projects. Activity awareness is a preconditionfor effective communication, planning, coordination,decision-making, and actions during long-termcollaboration.

With respect to the other concerns about awareness inCSCW research (e.g., workspace and social awareness),activity awareness is articulated at a higher level ofanalysis. Thus, it has a larger scope since it pertains to theunderstanding of the overall activity being performed in thecollaboration. Drawing from the conceptualization ofactivity in activity theory [18], we consider an activity as asequence of actions, directed towards a goal or object,mediated by tools, and situated in many embedded contexts(e.g., work practices, culture, organizational structures,interpersonal relations). Activity awareness pertains togroup activity that takes place over an extended period oftime. This implies that in order to maintain awareness of theentire activity, the group members need to develop andmaintain common understanding of shared goals, plans,norms and roles; monitor the resources over time; andremain aware of the actual status of the execution of thegroup activity and its relationship with the prior aspects.

Little empirical research has been conducted on activityawareness. Several authors have proposed differentcategories of awareness: peripheral, situational, informal,group, social, workspace, etc. However, in order toefficiently support synchronous and asynchronouscollaboration within long-term planned activities, we need abroader view of awareness that interprets the multiplecategories of awareness and their evolution in the largercontext of activities. Because activity awareness implies anextended collaboration process, it was an ideal candidatefor exploring our new laboratory methods. In this initialinvestigation, we began by modeling and studying activityawareness breakdowns observed from a field setting [4].

MODELING REAL-WORLD CONTEXTSThe field study that grounded our laboratory work involvedinter-classroom collaboration in a long-term school project.Small groups of middle school students, typically 3 studentsusing one computer, were paired across two differentclassrooms (6th and 8th grades) and collaborated remotely on

a science project. The remote collaboration was supportedthrough the BRIDGE software, a Java-based collaborativesystem [11], which provided the users with planning tools(calendar and timeline) and a collaborative multimedianotebook [4]. A multifaceted evaluation framework wasadopted to identify factors that disrupted or contributed toactivity awareness, and data was collected about the processand outcomes of collaboration [22].

We analyzed our field study records to categorize thecircumstances under which activity awareness breakdownsoccurred. On this basis, we developed a set of collaborativescenarios to be used in our laboratory study. For example,we found that collaborative breakdowns are distributedacross collaborators and contexts and often involve multiplepeople and actions to identify and repair. They inherentlyinvolve events with interdependencies and multipleconsequences for different collaborators. Situation, group,task, and tool factors were identified as four fundamentalcategories classifying activity awareness problems.Leveraging this framework and its ability to identifyactivity awareness problems occurring in similar contexts,we modeled the scenarios used in this research on thesefour factors.

A total of seven scenarios were used in the lab study (Table1). Each involved the participation of a confederate [7, 33]who was a trained research assistant playing the part of aremote collaborator. In the first scenario, Tool Use, theconfederate simply encouraged the participant to use aplanning tool. Each of the remaining scenarios then“implemented” a typical awareness breakdown observed inthe field [4], and corresponded to one of the factorsidentified in the fieldwork analysis: situation, group, task,and tool (Table 1, second column). The scenario introducedthe breakdowns through a combination of instructions to theconfederate and changes made to the collaborativeenvironment (i.e., before beginning a given session).

Additional Work and Task Data Changes were determinedby alterations in the length and the content of the taskrespectively (task factors); Schedule Changes andCompletion Failure were motivated by changes in the classschedule and unavailability of internet connection(situational factors); Role Changes was implemented as thepartner’s tendency to work alone combined withuncertainty about the partner’s abilities (group dynamics);Tool Changes was determined by the individual decision ofputting data in a different tool (tool factors). Thesescenarios describe unexpected changes occurring withinspecific collaborative contexts.

315

THE LABORATORY METHODBased on the data collected from the field study, wedeveloped a method that allows simulating andmanipulating authentic collaborative situations in alaboratory setting. This method is characterized by threemajor properties:

1. The use of authentic tasks and collaborative situations2. The use of a confederate3. The use of multiple collaborative sessions over time

The first characteristic refers to collaborative scenarios thatwere developed from field observations. This aspect of thelab method directly supports ecologic validity of thecollaborative context and the realism of the tasks performedduring the collaboration. The second characteristic wasintroduced to reduce the sources of variability in thelaboratory setting and to control and manipulate the settingsystematically. The third characteristic was motivated bythe necessity to study realistic long-term activities and thefactors that influence the way these activities unfold. Byusing realistic long-term activities, we can investigate theevolution of complex phenomena (i.e. planning and activityawareness). In fact, long-term collaboration requires usersto be aware of state information about the workspace and

the shared plans that constantly change within and betweenthe collaborative sessions.

Participants and laboratory settingThe participants were six 7th and 8th graders (P1… P6):three females and three males. They participated in fourweekly laboratory sessions in which they had to collaboratethrough a CSCW system (described below, see Figure 1)with another student (the confederate) on a group project.During the four sessions they had to complete anenvironmental project (Environmental Quality by NeoSCICorporation). Each session lasted approximately one hour.

The laboratory setting is an important component of ourmethod. The participant and the confederate were located inseparate rooms. Both were able to talk to the experimenterthrough a microphone and a video camera, and they wereboth able to hear the experimenter through computerspeakers. The experimenter had visual and auditory accessto both participant and confederate, and was able to switchthe audio communication between the two. Through the useof four screens (two displaying the participant’s andconfederate’s collaborative workspace, and two showingtheir interaction captured from the video cameras), theexperimenter could give personalized instructions, taketime-stamped notes, and guide the sessions. Theexperimenter also had his own computer station running aGroove client (Figure 1). This allowed the experimenter toaccess the collaborative workspace and intervene whennecessary. The confederate and participant could not see orhear each other, and both were physically separated fromthe experimenter. In the context of a realistic collaborativesituation, this setting allows the experimenter to monitorand influence the participant or confederate individuallywithout impacting the other.

Figure 1. Groove workspace [12]. The major components ofthe user interface: buddy list (A), planning tool (B), the actual

set of tools hierarchically organized within a large tabbedpanel (C), and the chat tool (D). Several awareness features

are supported (e.g., small pop-ups and notifications inform theuser about partners' movements through the workspace or

when they are typing a new message (E)).

Scenario Breakdownfactor

What the confederate (C)does in the scenario

Tool UseTool factors:the planningtool is used

The confederate (C) encouragesthe use of a planning tool

AdditionalWork

Task factors:the task isextended

C completes additional workbecause of new teachers’instructions: 3 additional

vocabulary terms were added

ScheduleChanges

Situationalfactors: the

class schedulechanges

C changes the dates in theplanning tool: two dates werechanged in the planning toolbecause the class schedule

changes

CompletionFailure

Situationalfactors:

unavailabilityof the internet

connection

C fails to complete a taskbecause of local contingencies:additional information was not

gathered from the Web

RoleChanges

Groupdynamics: a

task is executedahead ofschedule

C executes a task ahead ofschedule because of his habit to

work alone and uncertaintywith the partner’s abilities

Task DataChanges

Task factors:the content of

the taskchanges

C executes a task because ofnew teachers’ instructions: thelevels of pollutant consideredwere different from what islisted in the activity guide

Tool Change

Tool factors:the task is

completed in adifferent tool

C completes a task in a tool thatis different from the one they

had previously agreed on

Table 1. Scenarios summarized by breakdown factor andconfederate’s activities.

A

DE

C

B

316

Tool and tasksThe participants collaborated on a long-term projectthrough Groove, a computer-supported collaborative tool.This is a groupware system that supports asynchronous andsynchronous collaboration through a shared workspace [12](Figure 1).

The group project included the following collaborativeactivities that were conducted in the Groove workspace:Getting Acquainted (responding to questions about personalexperiences with environmental problems); Identifying theProblem (identifying the overall goal of the project);Developing a Plan (planning the activities for the entireproject); Vocabulary (defining a small set of scientificterms); Research Questions (identifying the questions to beinvestigated); Web Research (collecting relevantinformation from the Web); Develop and Conduct two labstudies (collecting lab data about acid rain and water);Organize, Graph, and Analyze Data (presenting the datacollected in the laboratories studies and from the Web);Draw Conclusions and Final Report (reporting about thewhole project).

Based on the categorization of activity awareness problemsthat emerged from the fieldwork [4], we developed a set ofscenarios that modeled the circumstances in whichawareness breakdowns occurred (see the section “Modelingreal-world context”). The scenarios were simulated throughthe confederate, who followed loosely scripted activitiesduring the four collaborative sessions. The confederateplayed the role of a middle school student of the same ageand gender. The use of pairs of the same gender allowedmitigating any confounding dynamics that could occuramong males and females students of this age group.

Here we provide an example of a script used by theconfederate to simulate the scenario ‘Schedule Changes’.

Scenario: Schedule Changes. Because of a change in the teacher’splanning of class activities, the dates of two activities in the projectmanager have been changed.

The day after your meeting with your partner, your teacher has decidedto swap the order of two class activities. Both of these activities arerelated to the work you are doing in the project. Since she requires youto perform the project task in parallel with the related class activity, shehas asked you to adjust the plan about the lab activities. Following hersuggestion, you have changed the schedule regarding the two activitiesin the Project Manager.

Experimental procedureThis section describes the organization of the four sessions(Table 2). In the first session, after signing the informedconsent, the participant was informed that s/he was going towork with another middle school student who was locatedin a neighboring school. The participant then read adescription of the experimental procedure and a briefoutline of the project activities. After s/he was trained onhow to use the workspace, s/he was given a demonstrationon how to think-aloud during the session.

The tasks to be accomplished during the first collaborativesession included three activities: Getting Acquainted,Identifying the Problem, and Developing a Plan.

During the other three collaborative sessions, theparticipants and the confederate had to plan their work forthe week and perform the scheduled activities. In the timeinterval between the collaborative sessions, the participantsdid not have to actually do the work. Instead, they receivedall their work for each session when they arrived. Thissimulated the work and allowed for a greater level ofcontrol. The schedule followed during the four sessions issummarized in Table 2.

Sessions Activities within andbetween sessions

Scenarios

Session 1:

1. the participant received:

a) detailed informationabout the project,

b) basic training on how touse Groove workspace andthe think-aloud technique.

2. collaborative session

All participants wereexposed to the scenario ToolUse

betweensessions

The participant received thetasks already accomplishedby email and read it

The workspace was modifiedaccording to the scenariosrun in the second session.

Session 2:

1. the participant was askedto insert her/his work in theworkspace and then explorewhole the content

2. collaborative session

All participants but P1 wereexposed to the scenarios:Task Expands a n dSchedule Changes

betweensessions

The participant received thetasks already accomplishedby email and read it

The workspace was modifiedaccording to the scenariosrun in the third session.

Session 3:

1. the participant was askedto insert her/his work in theworkspace and then explorethe whole content.

2. collaborative session

All participants wereexposed to one or more ofthe scenarios CompletionFailure, Role Changes, andTask Data Changes

betweensessions

The participant received thetasks already accomplishedby email and read it

The workspace was modifiedaccording to the scenariosrun in the fourth session.

Session 4:

1. the participant was askedto insert her/his work in theworkspace and then explorethe whole content.

2. collaborative session

3. questionnaire, interview

All participants wereexposed to one or both thescenarios C o m p l e t i o nFailure and Role Changes.

after thefourthsession

The participant was paidand was informed about thesimulation

Table 2. Collaborative sessions and scenarios schedule. Thetable summarizes the schedule of activities performed (secondcolumn) and scenarios run (third column) by session.

At the end of each session the participants were asked forinformal feedback regarding the recent session. At the endof the experiment, the participants filled out a questionnaireand participated in an interview. Finally, participants werecompensated $20 for their participation and were debriefed.

317

Before the experiment, the confederate was trained tosimulate the scripted scenarios using Groove, (i.e. theexperimenter acted as a subject and the confederate workedwith the system to complete the script). Moreover, beforeeach collaborative session the confederate reviewed thescripts for each scenario (Table 1) scheduled and thendiscussed with the experimenter how to flexibly adapt thescripts to the specific participant. Except for the firstscenario, each scenario was scheduled in accordance withthe plan of the activities defined by the participant andconfederate dyad (see third column on Table 2).

Data collection and analysisMultiple data collection techniques were used. During eachsession, the interaction between participant and confederatewas synchronously monitored and recorded using videocameras and a screen-capture tool. Both participant andconfederate used the think-aloud method to inform theexperimenter about what was occurring during the session.A session-by-session logging captured changes to theworkspace and the tasks assigned. The confederate alsomade notes during the experiment. Finally, a smallquestionnaire (Table 3) was given and a semi-structuredinterview was administered at the end of the last session.We also used contextual inquiry to interview theparticipants during the collaborative sessions.

We conducted three different types of analysis on the datacollected: analysis by scenarios, questionnaire andinterview, and breakdown analysis.

In the analysis by scenario, the participant’s activityawareness was assessed with respect to the changes (relatedto situation, people, task, and tool) occurring in eachcollaborative scenario. Two judges, both graduate HCIresearchers, conducted the assessment for the level ofawareness in participants by adhering to the followingcoding scheme:

1. Participants were evaluated ‘fully aware’ when theyhad spontaneously noticed the inconsistencies.

2. They were evaluated ‘partially aware’ if they noticedthe inconsistencies after being prompted by theconfederate or the experimenter.

3. They were considered ‘unaware’ in all remainingcases.

In the case of ‘fully aware’, the participant directly (e.g.through explicit statements) or indirectly (e.g. throughrelated comments or actions) showed that s/he wasconscious of the changes that occurred in a specificscenario. In the second and third cases, the participant didnot become spontaneously aware after being exposed to thechange. It was only after s/he moved on to another activitythat was unrelated to the scenario, that s/he was given oneprompt (through a comment or a question) from either theexperimenter or the confederate (directed by theexperimenter). If the participant then provided any direct or

indirect signs of being aware then s/he was evaluated as‘partially aware’. Otherwise, the participant was considered‘unaware’.

The questionnaire (Table 3) contains thirteen statementsabout activity awareness using a 7-point Likert-type scalemodeled on the rating scales proposed by Watts et al. [31].A follow-up semi-structured interview was also used tocollect qualitative data from the questionnaire(interpretation and reasons for the answers to each item)and to gather some additional issues that had emergedduring the experiment.

The breakdown analysis was conducted with explicitdefinitions of breakdown and critical incident. Acollaborative breakdown occurs in an interaction when theexpectations of one participant do not match with the actionof another [34, 8]. Partially overlapping with this concept,we consider critical incidents as behaviors and experiencesleading to surprisingly bad or good results [10].Consistently with the evaluation framework used for theanalysis of the breakdown in fieldwork [4], we consideredthat these breakdowns might be determined by differentclasses of factors: situational (environment), group (usersand their roles), task (plans), and tool (tools andworkspace). Using this evaluation framework, the twojudges analyzed the communication transcripts to identifyinstances of collaborative breakdowns and critical incidentsand to classify them according to the framework. Then thejudges compared, discussed, and agreed upon each case.

RESULTS: A VALID LABORATORY METHOD OFACTIVITY AWARENESSIn this section we present a collection of results that supportthe validity of this laboratory method for studying activityawareness. The validity of the method is assessed bycomparing the results of our study (collected through thethree types of analyses) to what we observed in the field. Ifthe results of this method are representative of what occursin the field, then the initial stages of validating the methodwill have been achieved. From the scenario analysis, weobserved several attributes of real collaboration occurringacross the four sessions. The participants appeared visiblyinterested in the activities and generally motivated to workwith their partner (only one participant was noticeably lessinterested in the topic, but this appeared to be due topersonal factors of the participant). For example, inresponse to the Tool Use scenario, all the participantsagreed to use one of the planning tools, and continued touse it during the remaining sessions. The participants didnot need to be prompted to keep using the planning toolsafter being exposed to the Tool Use scenario, demonstratingthat they were highly engaged over the four sessions.

In all the remaining scenarios, several activity awarenessproblems were observed. In more than half of the situations(18/31), participants did not become aware of the changesintroduced by the scenarios. Even among the situations inwhich participants were aware (13/31), only in 30% of the

318

situations (4/13) were changes noticed after being promptedby the experimenter or the confederate. In severalsituations, the participants were not fully aware of thechanges made to the content, workspace, and the tasks thatthey had agreed to perform.

Other awareness problems appeared to be caused by thelack of a clear overview of the shared plan. In fact, inseveral cases participants appeared to not fully understandthe duration of tasks, and tended to underestimate the timeneeded for the whole project. They tended to refine theirplans more during multiple sessions, negotiating decisionswith their partner as the work unfolded. For example, P1,P3, and P6 kept readjusting their plan until the third orfourth session. Defining a clear shared plan during thecollaboration and maintaining a constant awareness of theplan and time needed for each activity are difficult in realcollaborations, and the results from our laboratory methodconfirm this. These activity awareness difficulties assessedduring the experiment appear representative of onesobserved in real collaborative contexts and with theoreticaccounts for the importance of opportunistic aspects of theplanning activity [17, 30].

Examples A. Negotiation and collaborative problem solving

• P1 discusses with the partner how to split the work and how torealize each activity. Later, he will propose to assign a priorityvalue to each one of them.

• P3, after noticing that they did not have enough time to completethe project, invites the partner to re-examine the plan.

• P5 actively debates with her partner about the organization of thefinal report until they reach an agreement.

• P6 evaluates with the partner about the time to be assigned to thefinal report.

Examples B. Creativity

• P1, when adding the tasks to the planning tool, proposes todistinguish and assign different roles so that one can add the tasksto the tool and the other evaluates the accuracy and suggestschanges.

• P3 and P5 decided to use both Calendar and Project Manager,and, in order to transfer the data from one tool to another, theystrategically coordinated the work of the pair so that one of themwould read and type in the chat the tasks and the other wouldinsert the tasks into the planning tool.

Another aspect that confirms the validity of our method isthat during the collaborative sessions, the participantsengaged in lively negotiation and collaborative problemsolving with their partner (see examples above). We alsoobserved several cases in which the pairs of collaboratorsaccomplished some tasks creatively by defining their ownstrategy (see examples above). This shows that theexperimental procedure was flexible enough to allow thepair to organize the work creatively, as students are oftenencouraged to do in school contexts.

Finally, the analysis of the different data collected duringthe interaction (videos, the chat history, and think-aloud

verbalizations) showed that the level of engagement of theparticipants clearly increased when they had non-taskrelated communication with the partner. For example, withP6 the confederate engaged in a conversation about HarryPotter and with P5 about a recent movie. When theconfederate promoted more interpersonal and informalcommunication with three participants, they appeared toenjoy the opportunity to communicate informally and weremore engaged in the activities. Being more familiar withtheir partner seemed to help.

Questionnaire Av. s.d.

1. I found it difficult to tell what work my partner haddone after being absent from the workspace for a week.

2.8 1.7

2. It was easy to find what my partner had worked on inthe collaborative space.

5.0 1.5

3. I could tell what my partner was doing while wewere collaborating online.

6.2 0.8

4. I always knew what my partner was going to work onover the week.

6.3 0.8

5. It was always clear what my partner was going to do. 5.7 0.5

6. I became more aware of my partner’s plans overtime.

6.2 1.2

7. My partner and I planned adequately. 5.8 0.8

8. My partner and I communicated well with eachother.

6.5 0.5

9. My partner collaborated with me to complete theproject.

6.5 0.5

10. My partner contributed equally to this project. 7.0 0.0

11. I enjoyed collaborating with a partner online. 6.5 0.5

12. I would enjoy interacting with others in thecommunity (outside of the school system with interestor knowledge in science) on my group science project.

4.8 1.8

13. I would prefer to work on group projects over othertypes of school learning activities.

4.2 1.5

Table 3. Questionnaire results with average rankings andstandard deviation. The 7-points Likert-type scales range

from: 1 = Strongly Disagree to 7 = Strongly Agree.

These results are also confirmed by data from thequestionnaire (see Table 3) and the interview. Specifically,the results from the questionnaire show that participants feltthey were collaborating during the experiment, enjoyed theexperience, and were satisfied about the work and theirpartner. Moreover, the same participants who had engagedin non-task related communications appeared to enjoy thecollaboration with their partner more. During the interviewthey explicitly stated that communicating informally withtheir partner had made the other person more familiar andintimate (a student like them), and created a more realisticand informal context of collaboration. This supports theecologic validity of the method since in the real worldsocial behavior and work are always interleaved.

319

Using the evaluation framework defined in the analysis ofthe breakdowns within the prior fieldwork (see sub-section“Experimental procedure”), we conducted both a qualitativeand a quantitative analysis of the data about breakdownsextracted by the two judges. We found that breakdowns andcritical incidents occurred during all four sessions and didnot appear to be directly related to the manipulationintroduced through the scenarios. For example, in the firstsession, although the scenario Tool Use did not expose theparticipants to any inconsistencies, at least one third of thetotal breakdowns occurred within this session.

The analysis of the breakdowns by category has shown thatin more than one third of the cases (37%) the breakdownsoccurred because of problems related to communication,roles, and the relationship between partners. About anotherthird (32%) of the breakdowns were determined by taskfactors. Less that one fourth (23%) were related to toolfactors and a small portion (7%) were caused by situationalfactors. We observed that the number of cases ofbreakdowns tended to decrease along the four sessions(44%, 22%, 19%, 15%). This trend is easily explained bythe fact that the participants gradually became morefamiliar with the tool, the task, and the partner.

The problems related to communication, roles andrelationship between partners have been well acknowledgedby studies of computer mediated communication in realsettings, and have been considered to be a consequence ofthe restrictions imposed by the medium of communication(e.g., limited support for non-verbal communication,reduced number of auditory cues, deictic and spatial co-references that are difficult to resolve). Collaborators needto constantly repair or remediate miscommunications andundertake explicit actions to maintain common ground [5]and reciprocal awareness among the collaborators.

The category of breakdowns related to tasks and plansappeared strictly associated with participants’ problems inactivity awareness. Several breakdowns that we identifiedas part of this category occurred when participants noticedthat their plan was inappropriate because variousconstraints (e.g., duration or order of the activities) had notbeen considered (P1, P3). Moreover, in some conditions theparticipants were not able to predict what they needed to donext, and they appeared confused (P4, P6). These types ofbreakdowns reveal that in these situations the participantswere unable to maintain a clear overview of the plan and ofthe time available, and were not fully aware of the currentstatus of the work. This essentially reveals that they werelacking activity awareness.

The results obtained through the three different types ofanalysis show that this laboratory method was a validmodel of real collaboration. In fact, this method was able topromote engagement, lively discussion, autonomousinitiatives, collaborative problem-solving, and activityawareness difficulties, which are representative attributes of

real collaboration, and that were also observed in fieldstudies conducted in similar conditions.

DISCUSSION AND CONCLUSIONIn the previous section we presented data and arguments insupport of the validity of our laboratory method. Ourpurpose was to provide evidence of ecological validity; thatis, the extent to which our results were comparable to whathappens in the “real world”. Integrating the data from thethree different types of analyses shows that severalattributes of real collaboration were exhibited during theexperimental simulations: engagement, lively discussion,autonomous initiatives, collaborative problem solving, andactivity awareness difficulties.

This method also has provided us with relevant informationabout activity awareness. It allowed us to assess theparticipant’s level of activity awareness within thecollaborative scenarios simulated through the confederate.

Moreover, by systematically manipulating the collaborativeconditions, we were able to identify possible factors thataffect participant’s activity awareness. Our analysis byscenario has shown that the participants’ level of awarenessvaried for the different scenarios and across the foursessions. The difference between scenarios is particularlyevident if we compare the scenario Completion Failure andAdditional Work. While all the participants were aware ofthe inconsistencies that occurred in the Completion Failure,none of them were aware of the breakdowns introducedduring the Additional Work scenario. This differencesuggests that participants were generally more aware ofmacroscopic changes occurring to objects in the workspace(Completion Failure, Role Changes, and Tool Changes)rather that of changes of “smaller granularity” occurring tosymbols (Additional Work, Task Data Changes) within theworkspace. Another possible factor affecting participants’level of awareness is the growing familiarity with theworkspace, partner, and tasks during the experiment. Wealso observed that as participants advanced through the foursessions there was an increase in the number of cases ofparticipants’ awareness of said inconsistencies. This findingwas confirmed by all three type of data analysis. Forexample, the breakdown analysis had shown that thenumber of cases of breakdowns tended to decrease alongthe four sessions.

Based on our systematic manipulation of the experimentalconditions, we observed an increased trend in the subjects’level of activity awareness over the four sessions. Thistrend can be explained by general explanatory conceptsused in HCI and CSCW such as expertise, familiarity [15],and increasing common ground [5]. However, futureinvestigations are needed to evaluate the specific role thatthese factors play in activity awareness. Furtherinvestigation of these issues will require settings wherefactors such as familiarity and common ground can bemanipulated. Since these background factors are difficult tocontrol in field studies, a valid and cost-effective method

320

for research in this area would be the laboratory method wehave presented.

Having the ability to design, reproduce, and systematicallymanipulate collaborative situations in semi-naturalisticsettings means researchers can develop ad hocimplementations of this method suited to study complexcollaborative phenomena. This can be accomplished byusing a limited investment of resources and time. Themethod of manipulating the experimental setting and usingcollaborative scenarios makes this method categoricallydifferent from field studies, yet complementary to them. Forexample, contextual factors cannot be controlled within thefield studies, but they may have very complex influences onthe variable of measurement in the field. At the same timemanipulation is possible in a lab simulation, but thecontextual factors would be modeled using criteria andresults derived by empirical studies in the field. Inagreement with other authors [1, 32], our research paradigmto study collaboration combines methods from differentresearch traditions in order to provide richer and broadercontributions to our understanding of groups as complexsystems. In fact, in our line of research we investigateactivity awareness by interleaving lab and field studies.

The use of a multiplicity of data collection techniques in anintegrated fashion allows maximization of the amount ofknowledge extracted from the phenomena investigated bytriangulating or mediating [3, 28] multiple types of data andinterpretations. In our study, on several occasions, we wereable to integrate and triangulate data from the different datacollection techniques to increase our understanding of thedata and test the validity of the method. For example, thefact that the participants’ behavior evolved along the foursessions was confirmed by different types of data: theincreasing trend in the number cases of participants’awareness of inconsistencies and the decreasing trend in thenumber of breakdowns occurred.

Our results show the possibility for more articulatedinvestigations of the contrasting impacts that contextualfactors have on people’s overt and covert behaviors.Specifically, we observed that in some cases the datacollected through direct observation of overt behaviorswere inconsistent with the participants’ individualexperiences. Through alternative active investigationtechniques (through contextual inquiry, think-aloudprotocol and interviews), we were able to investigateunderlying experiences of the participants (e.g., personalreaction to the partner’s behavior) that would not beobservable from their overt behavior during the session. Anexample of this phenomenon was P2’s reaction to thescenario Role Changes, where the confederate hadcompleted additional work without any prior agreementwith her. During the collaborative session with her partner,she appeared pleased that her partner had accomplishedsome extra work ahead of time. However, after the session,when she was asked to comment on this specific situation,she expressed that she had felt disappointed. These types of

inconsistencies would have been difficult to study in thereal world. People, in fact, tend to conform to the normsand constraints of the context (e.g., social rules); forexample, the participants avoided to express personalconcern about their collaborators and their work (e.g. seeresults for the 10th item of the questionnaire, Table 3).

Although we designed and implemented the method tostudy activity awareness, this laboratory method might alsobe used to study other aspects of collaboration arising inreal contexts and during extended collaboration activities.For example, from our data analysis we were able toobserve phenomena related with social and actionawareness, planning, common ground, familiarity, learning,etc. We believe that this method may offer a newopportunity to study these phenomena in the laboratoryusing valid methods.

FUTURE WORKFuture work should focus on the research method, the datacollection, the data analysis, and the domain of study. Toimprove our research method in the future, specifically thecollaborative setting, we plan to increase engagement andecological validity: increase realism by making theparticipants actually complete the assigned work, which hasa cost-benefit tradeoffs; construct multiple sessionscenarios, which would assent to observing how scenariosaffect collaboration in the long term; and, encourageinformal communication between participant andconfederate before and after each session, which canbecome a motivational factor that induces engagement andintimacy. Another area that has potential for improvementis the think-aloud method, specifically as a formative datacollection technique for activity awareness. By asking theparticipants to think-aloud about what they think theirpartner knows, we could increase our ability to monitortheir awareness knowledge level. Our study also allows forfuture work in the area of awareness-sensible designs forCSCW systems within education. Particularly, the issues ofhuman-computer interaction associated with middle andhigh school students’ use of technology and theirdeveloping meta-cognitive and meta-communicationabilities could be further developed.

REFERENCES1. Arrow H., McGrath, J. E. Berdahl, J. L., Small Groups

as Complex Systems: Formation, Coordination,Development, and Adaptation. Sage Publications, 2000.

2. Baker, K. Heuristic Evaluation of Shared WorkspaceGroupware based on the Mechanics of Collaboration.MSc Thesis, Department of Computer Science,University of Calgary, Alberta, Canada, 2002.

3. Campbell, D., & Fiske, D. W. Convergent anddiscriminate validation by the multitraitmultimethodmatrix. Psychological Bulletin 4, (1959), 297-312.

4. Carroll, J. M., Neale, D. C., Isenhour, P. L., Rosson, M.B., and McCrickard, D. S. Notification and awareness:

321

synchronizing task-oriented collaborative activity. Int..Journ.. of Human-Computer Studies 58, (2003) 605-32.

5. Clark, H. H. Using Language. New York: CambridgeUniversity Press, 1996.

6. Cockton, G., & Woolrych, A. Sale must end: Shoulddiscount methods be cleared off HCI's shelves?,interactions 9(5) (2002), 13-18.

7. Connolly, T., Jessup, L., & Valacich, J.S., "Effects ofanonymity and evaluative tone on idea generation incomputer-mediated groups," Management Science 36,(1990), 689-703.

8. Easterbrook, S. (ed.), CSCW: Cooperation or Conflict,Springer-Verlag, New York, 1993.

9. Endsley, M. R., Toward a theory of situation awarenessin dynamic systems. Human Factors 37, (1995), 32-64.

10. Flanagan, J. C. 1954. The critical incident technique.Psychological bulletin 51(4), (1954), 327-358.

11. Ganoe, C. H., Somervell, J. P., Neale, D. C., Isenhour,P. L., Carroll, J. M., Rosson, M. B., and McCrickard, D.S. Classroom BRIDGE: Using Collaborative Public andDesktop Timelines to Support Activity Awareness.Proc. of the ACM '03 Symposium on User InterfaceSoftware and Technology, ACM Press, (2003) 21-30.

12. Groove Networks Inc., http://www.Groove.com, 2003.

13. Grudin, J., Why CSCW Applications Fail: Problems inthe Design and Evaluation of Organizational Interfaces,Proc. CSCW'88, (1988), 85-93.

14. Gutwin, C. & Greenberg, S. (2000). The Mechanics ofCollaboration: Developing Low Cost UsabilityEvaluation Methods for Shared Workspaces. IEEE 9thInt’l Workshop WET-ICE'00 (2000).

15. Gutwin, C. & Greenberg, S., Workspace AwarenessPosition paper for the ACM CHI'97 Workshop onAwareness in Collaborative Systems, organized byMcDaniel S. E., & Brinck T., Atlanta, Georgia (1997).

16. Gutwin, C. & Greenberg, S., Workspace awareness forgroupware. Proc. CHI’96, ACM Press, (1996),208–209.

17. Hayes-Roth, B. & Hayes-Roth, F. A cognitive model ofplanning. Cognitive Science, 3, (1979), 275-310.

18. Kuutti, K. & Arvonen, T. Identifying potential CSCWapplications by means of Activity Theory concepts: Acase example. Proc. CSCW '92, ACM Press, (1992),233-240.

19. McGrath, J. E., Methodology matters: Doing research inthe behavioral and social sciences. In R. M. Baecker, J.Grudin, W. A. S. Buxton, & S. Greenberg (Eds.),Readings in human-computer interaction: Toward theyear 2000, Morgan Kaufmann, (1995), 152-169.

20. Mohammed, S., & Dumville, B., Team Mental Models:Expanding Theory and Measurement Through Cross-

Disciplinary Boundaries. Journal of OrganizationalBehavior, 22 (2001), 89-106.

21. Neale, D. C., Carroll, J. M., Rosson, M. B., EvaluatingComputer-Supported Cooperative Work: Models andFrameworks, Proc. CSCW ’04, ACM Press, (2004) (toappear).

22. Neale, D.C., Carroll, J.M., 1999. Multi-facetedevaluation for complex, distributed activities. Proc. ofthe CSCL’99 Computer Supported CooperativeLearning. Lawrence Erlbaum, 425–433.

23. Pinelle, D., & Gutwin, C., Groupware walkthrough:Adding context to groupware usability evaluation. Proc.2002, ACM Press, (2002) 455–462.

24. Pinelle, D., Gutwin, C., & Greenberg, S., Task Analysisfor Groupware Usability Evaluation: Modeling Shared-Workspace Tasks with the Mechanics of Collaboration,ACM Transactions on Computer-Human Interaction,10, (4) (2003), 281-311.

25. Prinz W., Mark G., Pankoke-Babatz P. (1998),Designing Groupware for Congruency in Use, Proc.CSCW'98, ACM Press, 373-382.

26. Ramage M., The Learning Way: Evaluating Co-operative Systems, PhD thesis, Lancaster Univ., 1999.

27. Salas, E., Prince, C., Baker, D.P., & Shrestha, L.,Situation awareness in team performance: Implicationsfor measurements and training, Human Factors, 37(1995), 123-136.

28. Scriven, M. The methodology of evaluation. In R. Tyler,R. Gagne & M. Scriven (Eds.), Perspectives ofcurriculum evaluation. Chicago: Rand McNally, p.39-83, 1967.

29. Steves, M., Morse, E., Gutwin, C., & Greengerg, S., Acomparison of usage evaluation and inspection methodsfor assessing groupware usability. Proc. 2001International ACM SIGGROUP Conference onSupporting Group Work. ACM Press (2001), 125–134.

30. Suchman, L.A. Plans and situated actions. New York:Cambridge University Press, 1986.

31. Twidale, M. Randal, D., Bentley, R. (1994), SituatedEvaluation for Cooperative Systems. Proc. CSCW'94,ACM Press (1994), 22-26

32. Watts, L., Monk, A., & Daly-Jones, O. Inter-PersonalAwareness and Synchronization: Assessing the Value ofCommunication Technologies. International Journal ofHuman-Computer Studies, 44 (6), (1996), 849-873.

33. Weisband, S., Overcoming social awareness incomputer-supported groups: Does anonymity reallyhelp?, Proc. CSCW’94 (2) (1994), 285-297.

34. Winograd, T., & Flores, F., Understanding computersand cognition: A new foundation for design. Norwood,NJ: Ablex, 1986.

322