Upload
irene-angelica-chounta
View
96
Download
2
Tags:
Embed Size (px)
DESCRIPTION
In this paper, we propose a method for the real time evaluation of collaborative activities. The method aims to classify collaborative activities using a memory-based learning model and to evaluate them using a reference set of pre-evaluated activities. The proposed approach uses time series to represent activities and classifies them regarding their similarity. In order to evaluate the results, the method is used as an automatic rater of collaboration quality and the ratings are compared to ratings of human experts.
Citation preview
It's all about time: towards the real time evaluation of
collaborative activities
Irene-Angelica Chounta, Nikolaos Avouris{houren, avouris}@upatras.gr
HCI Group, University of Patras
Real time evaluation
Why...● To support teachers
orchestrate classrooms (monitoring)
● To support awareness and provide feedback to teams (mirroring)
How...● Multiple views of
common workspaces/dialogues
● Metrics of interaction, statistics of participation & graphs
● Insight of work progress via comparison
Objective● To evaluate the quality of collaboration using
time series classification – User interaction represented as time series– Evaluation through classification– Activities are classified with respect to time series
similarity
● To study the effectiveness of the method with respect to the duration of the activity
Related work
● Time series classification was used for the post-evaluation of collaboration quality (Chounta, Avouris 2012)
● Activity metrics that correlate with collaboration quality used for time series construction of activity
● No need for hand-coding or specialized data manipulation
Collaborative setting● Dyads of students – 210
sessions● Duration of the activity:
90 minutes● Task: joint construction
of algorithmic flowcharts over a common workspace
● Synchronous communication over a chat tool
Real-time evaluation method
● A memory based learning model (TSCMoCA) was used as an automatic rater of collaboration quality:
Real-time evaluation method● The model TSCMoCA is called on demand by
the teacher
● An evaluative value for the quality of collaboration CQA is returned
Quality of collaboration● The quality of collaboration is defined on six
dimensions:● Collaboration flow● Sustaining mutual understanding ● Knowledge exchange ● Argumentation ● Structuring the problem solving process ● Cooperative orientation
● The collaborative activities were evaluated by human experts (Kahrimanis, Chounta, et.al 2009)
● The evaluative value of collaboration quality ranged between [-2, +2]
Communication
Joint information processing
Coordination
Interpersonal Relationship
Method of the study● Real time evaluation of collaboration quality
on 3 time instances:– 25 minutes ( ≈1/4 of total duration)
– 40 minutes ( ≈1/2 of total duration)
– Total Duration (≈90 minutes)
● Real time evaluation on six collaborative dimensions, after the first half of the activity
● The results of the automatic rater are compared with the ratings of human experts
Results (1)
25 minutes 40 minutes Total duration
Correlation (p<0.05)
0.27 0.51 0.59
Mean Absolute Error
0.99 0.94 0.91
● In all cases, the ratings of the model correlate significantly with the ratings of human experts
● Weak correlations for the case of 25 minutes● Big improvement for the case of 40 minutes● Similar results for the evaluation on total
duration
Results (2)
Collaborative Dimensions Correlation MAE
Collaboration Flow 0.53 0.92
Sustaining mutual understanding 0.42 1.02
Knowledge exchange 0.5 1.17
Argumentation 0.44 1.25
Structuring the Problem Solving Process 0.46 1.17
Cooperative orientation 0.47 1.07
● In all cases, the ratings of the model correlate significantly with the ratings of human experts
● The method performs effectively for the dimensions that reflect communication on a low level
● Low correlations/high error for complicated constructs or dimensions that reflect collaboration on the long run
Results (2)
Collaborative Dimensions Correlation MAE
Collaboration Flow 0.53 0.92
Sustaining mutual understanding 0.42 1.02
Knowledge exchange 0.5 1.17
Argumentation 0.44 1.25
Structuring the Problem Solving Process 0.46 1.17
Cooperative orientation 0.47 1.07
● In all cases, the ratings of the model correlate significantly with the ratings of human experts
● The method performs effectively for the dimensions that reflect communication on a low level
● Low correlations/high error for complicated constructs or dimensions that reflect collaboration on the long run
Results (2)
Collaborative Dimensions Correlation MAE
Collaboration Flow 0.53 0.92
Sustaining mutual understanding 0.42 1.02
Knowledge exchange 0.5 1.17
Argumentation 0.44 1.25
Structuring the Problem Solving Process 0.46 1.17
Cooperative orientation 0.47 1.07
● In all cases, the ratings of the model correlate significantly with the ratings of human experts
● The method performs effectively for the dimensions that reflect communication on a low level
● Low correlations/high error for complicated constructs or dimensions that reflect collaboration on the long run
Conclusions
● Real-time evaluation of collaboration by classifying activities with respect to their time-series similarity
● Adequate performance when the evaluation is carried out on about half of the duration
● The method can be used to evaluate low-level collaborative aspects unlike more advanced constructs
Discussion and future work● The proposed method was studied for a
specific collaborative setting (dyads/synchronous communication/collaborative diagram building)
● Generalization of the findings for a wider range of collaborative activities
● Application of the method in a real-classroom for teacher support
Thank you
Time Series Construction
Metric of Activity
#messages or #workspace actions
rate of #messages or #workspace actions
#role alternation
Rate of #role alternation17
Memory based learning model TSCMoCA
Time series of Collaborative Activity x
Training set
DTW similarity comparison of time series
Similarity Matrix
Classification of Collaborative Activity x with respect to KNN algorithm
TSCMoCA – Teacher Interface