19
It's all about time: towards the real time evaluation of collaborative activities Irene-Angelica Chounta, Nikolaos Avouris {houren, avouris}@upatras.gr HCI Group, University of Patras

It's all about time: towards the real time evaluation of collaborative activities

Embed Size (px)

DESCRIPTION

In this paper, we propose a method for the real time evaluation of collaborative activities. The method aims to classify collaborative activities using a memory-based learning model and to evaluate them using a reference set of pre-evaluated activities. The proposed approach uses time series to represent activities and classifies them regarding their similarity. In order to evaluate the results, the method is used as an automatic rater of collaboration quality and the ratings are compared to ratings of human experts.

Citation preview

Page 1: It's all about time: towards the real time evaluation of collaborative activities

It's all about time: towards the real time evaluation of

collaborative activities

Irene-Angelica Chounta, Nikolaos Avouris{houren, avouris}@upatras.gr

HCI Group, University of Patras

Page 2: It's all about time: towards the real time evaluation of collaborative activities

Real time evaluation

Why...● To support teachers

orchestrate classrooms (monitoring)

● To support awareness and provide feedback to teams (mirroring)

How...● Multiple views of

common workspaces/dialogues

● Metrics of interaction, statistics of participation & graphs

● Insight of work progress via comparison

Page 3: It's all about time: towards the real time evaluation of collaborative activities

Objective● To evaluate the quality of collaboration using

time series classification – User interaction represented as time series– Evaluation through classification– Activities are classified with respect to time series

similarity

● To study the effectiveness of the method with respect to the duration of the activity

Page 4: It's all about time: towards the real time evaluation of collaborative activities

Related work

● Time series classification was used for the post-evaluation of collaboration quality (Chounta, Avouris 2012)

● Activity metrics that correlate with collaboration quality used for time series construction of activity

● No need for hand-coding or specialized data manipulation

Page 5: It's all about time: towards the real time evaluation of collaborative activities

Collaborative setting● Dyads of students – 210

sessions● Duration of the activity:

90 minutes● Task: joint construction

of algorithmic flowcharts over a common workspace

● Synchronous communication over a chat tool

Page 6: It's all about time: towards the real time evaluation of collaborative activities

Real-time evaluation method

● A memory based learning model (TSCMoCA) was used as an automatic rater of collaboration quality:

Page 7: It's all about time: towards the real time evaluation of collaborative activities

Real-time evaluation method● The model TSCMoCA is called on demand by

the teacher

● An evaluative value for the quality of collaboration CQA is returned

Page 8: It's all about time: towards the real time evaluation of collaborative activities

Quality of collaboration● The quality of collaboration is defined on six

dimensions:● Collaboration flow● Sustaining mutual understanding ● Knowledge exchange ● Argumentation ● Structuring the problem solving process ● Cooperative orientation

● The collaborative activities were evaluated by human experts (Kahrimanis, Chounta, et.al 2009)

● The evaluative value of collaboration quality ranged between [-2, +2]

Communication

Joint information processing

Coordination

Interpersonal Relationship

Page 9: It's all about time: towards the real time evaluation of collaborative activities

Method of the study● Real time evaluation of collaboration quality

on 3 time instances:– 25 minutes ( ≈1/4 of total duration)

– 40 minutes ( ≈1/2 of total duration)

– Total Duration (≈90 minutes)

● Real time evaluation on six collaborative dimensions, after the first half of the activity

● The results of the automatic rater are compared with the ratings of human experts

Page 10: It's all about time: towards the real time evaluation of collaborative activities

Results (1)

25 minutes 40 minutes Total duration

Correlation (p<0.05)

0.27 0.51 0.59

Mean Absolute Error

0.99 0.94 0.91

● In all cases, the ratings of the model correlate significantly with the ratings of human experts

● Weak correlations for the case of 25 minutes● Big improvement for the case of 40 minutes● Similar results for the evaluation on total

duration

Page 11: It's all about time: towards the real time evaluation of collaborative activities

Results (2)

Collaborative Dimensions Correlation MAE

Collaboration Flow 0.53 0.92

Sustaining mutual understanding 0.42 1.02

Knowledge exchange 0.5 1.17

Argumentation 0.44 1.25

Structuring the Problem Solving Process 0.46 1.17

Cooperative orientation 0.47 1.07

● In all cases, the ratings of the model correlate significantly with the ratings of human experts

● The method performs effectively for the dimensions that reflect communication on a low level

● Low correlations/high error for complicated constructs or dimensions that reflect collaboration on the long run

Page 12: It's all about time: towards the real time evaluation of collaborative activities

Results (2)

Collaborative Dimensions Correlation MAE

Collaboration Flow 0.53 0.92

Sustaining mutual understanding 0.42 1.02

Knowledge exchange 0.5 1.17

Argumentation 0.44 1.25

Structuring the Problem Solving Process 0.46 1.17

Cooperative orientation 0.47 1.07

● In all cases, the ratings of the model correlate significantly with the ratings of human experts

● The method performs effectively for the dimensions that reflect communication on a low level

● Low correlations/high error for complicated constructs or dimensions that reflect collaboration on the long run

Page 13: It's all about time: towards the real time evaluation of collaborative activities

Results (2)

Collaborative Dimensions Correlation MAE

Collaboration Flow 0.53 0.92

Sustaining mutual understanding 0.42 1.02

Knowledge exchange 0.5 1.17

Argumentation 0.44 1.25

Structuring the Problem Solving Process 0.46 1.17

Cooperative orientation 0.47 1.07

● In all cases, the ratings of the model correlate significantly with the ratings of human experts

● The method performs effectively for the dimensions that reflect communication on a low level

● Low correlations/high error for complicated constructs or dimensions that reflect collaboration on the long run

Page 14: It's all about time: towards the real time evaluation of collaborative activities

Conclusions

● Real-time evaluation of collaboration by classifying activities with respect to their time-series similarity

● Adequate performance when the evaluation is carried out on about half of the duration

● The method can be used to evaluate low-level collaborative aspects unlike more advanced constructs

Page 15: It's all about time: towards the real time evaluation of collaborative activities

Discussion and future work● The proposed method was studied for a

specific collaborative setting (dyads/synchronous communication/collaborative diagram building)

● Generalization of the findings for a wider range of collaborative activities

● Application of the method in a real-classroom for teacher support

Page 16: It's all about time: towards the real time evaluation of collaborative activities

Thank you

Page 17: It's all about time: towards the real time evaluation of collaborative activities

Time Series Construction

Metric of Activity

#messages or #workspace actions

rate of #messages or #workspace actions

#role alternation

Rate of #role alternation17

Page 18: It's all about time: towards the real time evaluation of collaborative activities

Memory based learning model TSCMoCA

Time series of Collaborative Activity x

Training set

DTW similarity comparison of time series

Similarity Matrix

Classification of Collaborative Activity x with respect to KNN algorithm

Page 19: It's all about time: towards the real time evaluation of collaborative activities

TSCMoCA – Teacher Interface