It's all about time: towards the real time evaluation of collaborative activities

  • Published on

  • View

  • Download

Embed Size (px)


In this paper, we propose a method for the real time evaluation of collaborative activities. The method aims to classify collaborative activities using a memory-based learning model and to evaluate them using a reference set of pre-evaluated activities. The proposed approach uses time series to represent activities and classifies them regarding their similarity. In order to evaluate the results, the method is used as an automatic rater of collaboration quality and the ratings are compared to ratings of human experts.


<ul><li> 1. It's all about time: towards the real time evaluation of collaborative activities Irene-Angelica Chounta, Nikolaos Avouris {houren, avouris} HCI Group, University of Patras </li></ul> <p> 2. Real time evaluation Why... To support teachers orchestrate classrooms (monitoring) To support awareness and provide feedback to teams (mirroring) How... Multiple views of common workspaces/dialogues Metrics of interaction, statistics of participation &amp; graphs Insight of work progress via comparison 3. Objective To evaluate the quality of collaboration using time series classification User interaction represented as time series Evaluation through classification Activities are classified with respect to time series similarity To study the effectiveness of the method with respect to the duration of the activity 4. Related work Time series classification was used for the post-evaluation of collaboration quality (Chounta, Avouris 2012) Activity metrics that correlate with collaboration quality used for time series construction of activity No need for hand-coding or specialized data manipulation 5. Collaborative setting Dyads of students 210 sessions Duration of the activity: 90 minutes Task: joint construction of algorithmic flowcharts over a common workspace Synchronous communication over a chat tool 6. Real-time evaluation method A memory based learning model (TSCMoCA) was used as an automatic rater of collaboration quality: 7. Real-time evaluation method The model TSCMoCA is called on demand by the teacher An evaluative value for the quality of collaboration CQA is returned 8. Quality of collaboration The quality of collaboration is defined on six dimensions: Collaboration flow Sustaining mutual understanding Knowledge exchange Argumentation Structuring the problem solving process Cooperative orientation The collaborative activities were evaluated by human experts (Kahrimanis, Chounta, 2009) The evaluative value of collaboration quality ranged between [-2, +2] Communication Joint information processing Coordination Interpersonal Relationship 9. Method of the study Real time evaluation of collaboration quality on 3 time instances: 25 minutes ( 1/4 of total duration) 40 minutes ( 1/2 of total duration) Total Duration (90 minutes) Real time evaluation on six collaborative dimensions, after the first half of the activity The results of the automatic rater are compared with the ratings of human experts 10. Results (1) 25 minutes 40 minutes Total duration Correlation (p</p>