Collaborative projects and self evaluationwithin a social reputation-based

exercise-sharing system

Andrea SterbiniDept. of Computer Science

”La Sapienza” University, Rome, Italysterbini@di.uniroma1.it

Marco TemperiniDept. of Computer and Systems Science”La Sapienza” University, Rome, Italy

marte@dis.uniroma1.it

Abstract—We present the design issues and motivations of anenhanced version of the web-based system SOCIALX, supportingcollaborative and social aspects of learning. This web applicationallows to share solutions to exercises and development of project-(possibly group-) work, through the management of a reputationsystem. With the aim of enhancing collaboration and to helpstudents working on exercises, we introduce contextual FAQsand micro-forums and a currency-based concretization of theperceived usefulness of other’s answers. The tokens exchangedare used also to help the teacher/tutor in choosing the bestquestion/answer pairs to be promoted to the FAQ. To introducegroup responsibilities, peer-pressure and self-evaluation we de-fine group-based projects with self/peer-evaluated phases. Thedifferent phases of a project are given to different groups, sothat the produced deliverables are both self-evaluated when theyare submitted and peer-evaluated by the group working on thenext phase. The system is its last stages of development and willbe tested with real students in the next academic year.

I. INTRODUCTION

Cooperative learning is an indispensable element (in thee-learning field) to help learners sharing and combining ex-pertise, with the goal to prepare to join team-based workingenvironments [2]. Whereas cooperative learning is usuallydiscussed and applied on small groups, a further aspect ofinterest is then in the vision of e-learning as a communityand social activity [7]. Cooperative learning can improveteaching and learning considerably; moreover its implicationsand effects in the extension of present e-learning standard(namely the IMS Learning Design [4]) have just started tobe considered [8]. Fundamental didactic tools in collaborativelearning are reputation systems, that capture (and make evidentto the learner) the contributions s/he is giving to the group, tothe class and to the course. A reputation system is both a mo-tivational tool and a way to evaluate and understand learner’spsychological preferences, relations with others, ability toanalyze/judge others’ work, and thus conceptual competences.We are also working on a comprehensive approach to themanagement of personalized courses [3], [6], exploiting socialaspects of learning to enrich the definition of learner’s model.In particular, opposite to the usual approaches to collaborationbased on small groups, we are defining a model includingthe idea that learners are participating to a social network,

where they can interact, exchange information and collaborateover common problems (e.g. mandatory exercises in a subjectmatter).

In this paper we deal mainly with the designing aspects of aweb application, called SOCIALX, taking care of collaborativeand social aspects of learning in the aforementioned systemand allowing the management of an augmented learningmodel, based on the analysis of learner behavior in a specifi-cally devised reputation system environment.

II. THE OLD SOCIALX

Our previous system [5], paired a reputation system togetherwith an exercise-sharing web tool. It aimed at:• increasing the motivation of students in doing home-

works,• increasing/encouraging higher cognitive learning activi-

ties (as in the Bloom cognitive taxonomy [1]), both byrewarding the student grading other’s solutions, and byrewarding the reuse and correction of other’s solutions

To obtain this we built a reputation-based system, withinwhich a student was able to work on homeworks, sharehis/hers solutions and judge and reuse other’s solutions. A rep-utation system is normally used to motivate interaction and toelicit good behaviors by awarding points to the user’s actionsthat are deemed more useful to the community. In our case thestudent’s reputation is a blend of five facets that describe howwell s/he is working within the class: involvement, usefulness,competence, judgment, and critical thinking (see later fordetails). Reports of the student’s reputations can be shownboth at the course and topic level, with details displaying allthe facets to allow the student to improve his/her reputationby focusing on the type of social activity s/he likes more.

III. THE NEW SOCIALX

SocialX is being extended with the added goals of:• increasing collaboration and peer-based help by intro-

ducing contextual micro-forums within which ”directrewards” (tokens) are used to explicitly capture the per-ceived usefulness of other’s help,

• introducing peer-pressure and responsibilities towards thegroup by managing group-based projects.

2009 IEEE/WIC/ACM International Joint Conferences on Web Intelligence and Intelligent Agent Technologies

978-0-7695-3801-3/09 $26.00 © 2009 IEEE

DOI 10.1109/WI-IAT.2009.273

239

2009 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology

978-0-7695-3801-3/09 $26.00 © 2009 IEEE

DOI 10.1109/WI-IAT.2009.273

239

2009 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology

978-0-7695-3801-3/09 $26.00 © 2009 IEEE

DOI 10.1109/WI-IAT.2009.273

243

• support the teacher, which is a very valuable yet limitedresource, by exploiting to our best all the recorded socialinteractions among the intervening students.

Moreover, we introduce a new facet in the reputation systemto explicitly capture the self judgment ability of the student.

Definition 1: The reputation of a learner is an overallrepresentation of certain learner’s qualities as they come outfrom his/her interaction with the SOCIAL X system. It can becalculated at different levels of detail in the system: coursetopic, whole course, and whole system (encompassing severalcourses). There are six basic aspects that are taken care of inthe system:• involvement: the degree of active participation in the

system, measurable by the amount of work that thelearner has been available to submit, also in terms ofparticipation (such ad the number of solutions submitted,questions proposed and grades given, as well as thepropriety and extension of judgments);

• usefulness: how the learner’s work is beneficial to othersin the system (such as the reuse of learner’s solutions,and the appreciation of her/his questions);

• competence: an appraisal of the skills shown by thelearner (deriving from the grades and judgments comingfrom peer students and from the teacher;

• judgment: how well the student has evaluated other’ssolutions, questions, answers and products (with respectto the teacher’s grades and evaluations)

• self-judgment: how well the student has evaluated her/hisown answers and products (with respect to the teacher’sgrades and evaluations)

• active critical thinking: a measure of the conceptualwork issued to understand and critically appraise others’work, in order to modify, reuse, and start from suchwork (such as when a solution is the first produced for aproblem, or is the correction of another)

A. Increasing collaboration

We introduce both contextual micro-forums attached to eachexercise, so that students help each other by asking/answeringquestions, and FAQs to collect the most interesting discus-sions. The students exchanges are moderated by the teacher,that can ”promote” the discussion threads by refactoring themost interesting pairs of questions/answers to the exercise’sFAQ. When a discussion is refactored as a new FAQ entry, thestudents involved in the originating discussion are rewarded byincreasing their usefulness and competence reputation levels.

To make the best use of the teacher’s time, we highlight theexchanges that the students have already selected as the mostinteresting/appropriate.

B. Perceived usefulness

To enhance the motivation of the students in helping each-other, and to make explicit the perceived usefulness of otherswe apply the classical currency-based approach (tokens) thatstudents can use to acquire services by other students or bythe teacher. Each student is awarded an initial number of

tokens that can exchange with good answers. Whenever s/heneeds an information s/he can pose a question (consumingone token) and reward the best answer received. Tokens are alimited resource, and thus a student needing answers shouldfirst ”work” for the community to collect the tokens neededto ask more questions. The total number of tokens receivedis a direct indication of the usefulness of the student in thecommunity, and thus it contributes to the usefulness and tothe competence factors of his/her reputation. The number oftokens spent, instead, counts how many times the student hasasked questions to the community, and thus it contributes tothe involvement factor of his/her reputation.

To avoid students cheating the system (e.g. by exchanginguseless questions/answers) we mildly discourage ”off-topic”and ”dummy” discussions. The teacher/tutor flags this kind ofuseless exchanges so that they contribute zero to the reputationand the token spent to create the question is lost (for boththe students involved). Discouraged exchanges may affect thereputation of both parties involved.

Therefore the participation of the students at the contextualmicro-forums produces reputation through the rules:

1) the answers given to others (even if not awarded with thetoken) contribute to the student’s involvement factor,

2) the tokens received show how much a student has beenuseful to the others, increasing her/his usefulness factor,

3) the tokens spent to propose questions show how muchthe student has participated, and contribute to his/herinvolvement factor,

4) a Q/A promoted to FAQ shows that that contribution isimportant, thus contributing to the answering student’scompetence factor,

5) ”dummy” and ”off-topic” discussions are completelyignored and loose the corresponding token

To make the best use of the teacher’s time we highlight thetoken exchanges to help him/her to evaluate faster the dummyand FAQ candidates.

C. The teacher is the bottleneck

The teacher’s work in the system is a crucial, limitedresource. S/he should correct solutions, moderate answers,promote good Q/As to the FAQs, manage the group projects.We must make the best use of the teacher’s expertise, even ifs/he would be able to check/test/correct just a small part ofthe solutions submitted. To this aim we exploit the network ofsocial exchanges between the students to guide the teacher byselecting the most interesting items to be evaluated. Then, thesocial network is used to propagate the evaluation results to theneighbor items to adjust the authors’ reputations accordingly.E.g., the tokens exchanged in the contextual micro-forumsare used (also) to highlight the most useful answers, thusreducing the number of Q/As to peruse while looking forgood candidates for FAQ promotion. In the exercise evalu-ation the teacher is guided by the judgments expressed, theiragreement/disagreement, and the reputation of the interveningstudents.

240240244

IV. SOCIAL COLLABORATIVE PROJECTS

As seen, SOCIAL X allows the use of a reputation systemin an e-learning environment, supporting the development ofcollaborative-social exercising activities within a potentiallylarge group of students. Such “exercising activities” so havebeen made of single exercises, freely reusable by each learner.So, in the context of SOCIAL X the learning activity is a trade-off between individual work (selection and comprehensionof others’ work, reuse and adaptation, development of newsolutions) and social exchange.

However in certain courses the development of projects is arelevant part of learning, in both cases of an activity performedby the individual learner or a collaborative work carried onby a small group of learners. So we extended SOCIAL X toembed also the support to a partially social approach to thedevelopment of projects. The approach is called “partially”social collaborative because, while a small group (possiblysingleton) of learners is still the basic operating unit, theproducts of such units are submitted to social exchange withthe other units (to be reused and assessed). A project is usuallya prolonged and organized activity, made of a sequence oftasks, each one depending on previous one and dependedupon by the following ones. Usually a project is entrustedto a small group of learners, and collaborative work amongthem is instructed and supported, to produce the deliverablefor the whole project. We add to SOCIAL X the support toa partially social collaborative approach to the developmentof projects. Instead of having a small group working on thevarious steps of a single project, the idea is to have thegroup working on different steps of different projects: allthe projects share a similar structure, made of a sequence oftasks (the steps); the n-th task of a project is expected to be“similar” to the n-th task of another (wrt the general learninggoals related to the project development methodology); so thegroup would be assigned a path of tasks, each one possiblyinvolving a step in a different project; at each step the groupshould deliver a product; moreover, the learners in the groupprovide evaluations of the product(s) received from earlierstep(s) in the same project (from which the group shouldstart to work on its task) and of the deliverable released bythe group (to show self-evaluation skills). We define a socialcollaborative project (SCP), in a given course topic T , as aset of tasks PT = {ti}i∈(1,...,nT ). Each task is assigned toa group of learners (gT = {li}i∈(1,...,ngT )), that will do thecorresponding learning activity (such as the construction ofa deliverable product). Moreover, the sequence of tasks ina SCP provides a complete span of learning activities aboutthe related project methodology. (Henceforth, where possiblewe’ll assume that projects are all on the same T and avoidthe related indexes.)

In the following definition, an SCP-path is a sequence oftasks, selected from different projects in such a way to providethe aforementioned complete span of learning activities.

Definition 2: (work-field - WF - and SCP-path)

A WF is a set of projects {Pj}j∈(1,...,nWF )

A SCP-path in a WF is a set of tasks

{ti,j}i∈(1,...,nT ),j∈(1,...,nWF )

where ti,j is the i-th task in the j-th project of the work-field.

In a work-field, the projects are supposed to share a commonstructure, meaning that the number of steps and their logicalsequence are homogeneous, so that it is acceptable that a SCP-path provides group learners with a reasonably standard andcomplete project activity in the course topic. Once a suitablework-field is defined, SCP-paths can be assigned to groups.The following is an example of path assigned to a group g:

{t1,kg1

t2,kg2· · · tnT ,kg

nT} where

∀h ∈ (1 . . . nT ) kgh ∈ (1, . . . , nWF )

(the path is made by nT tasks (to make the overall activitycomplete according to the course topic definition); each i-thtask is the i-th task in one of the projects of the work-field).If we can assume that each task in a project depends on theprevious and is depended upon by the following, we can alsoassume that for almost each task undertaken by a group inits SCP-path, the group is going to depend on the work doneby other groups and will produce material for other groups touse. This gives the social dimension to the activities in a SCP-work-field, and gives also the opportunity to add feedbacksover the reputation of learners, beyond the evaluation of theirtechnical skills related to project deliverables. (In consideringthe dependences of a tasks from others, we limit the scopeto those immediately preceding and succeeding, in order tosimplify a bit the notation, with no prejudice for the generaldiscussion).

Definition 3: (fulfillment of a task by a group)Given a task ti,j assigned to group g = {lp}p∈Ig

, andassuming that the previous and successive tasks in the sameproject Pj , ti−1,j and ti+1,j , are resp. assigned to groups gand g, g fulfills ti,j when it provides the system with• A product p(g, ti,j),• A set of evaluations {VAL(lp, g, ti−1,j)}p∈Ig over the

product received from the previous task in the project(one explicit evaluation for each member of the group),

• A set of self-evaluations {AVAL(lp, g, ti,j)}p∈Igover

the product released by the group itself (one explicitevaluation for each member of the group),

When the group g = {lq}q∈Ighas fulfilled its task

ti+1,j another set of evaluations {VAL(lq, g, ti,j)}q∈Ig) will

be available over the work of group g.

So, from the work of a group of learners g in a SCP-work-field, and from Def.3, many items may produce a feedbackover the reputation of the group members:

1) for each task ti,j of the SCP-path assigned to g we havea set of evaluations of the product p(g, ti,j), issued bythe members of group g that followed g in the same

241241245

project Pj ; each grade, as well as the principal one givenby teachers, is imparted to the whole group and caneasily be spread, mediated by the teachers’ judgment, tofeedback over usefulness and competence of each lp ∈ g.

2) for each task, ti,j , we also have the evaluations issuedby group members about the product p(g, ti−1,j) inher-ited from the previous task in the same project: thoseare single learner’s evaluations, that can be comparedwith teachers’, affecting both learner’s competence andjudgment.

3) the various evaluations mentioned at point 1) are also tobe taken into account to measure the ability of groupg to build a good product, basing on the one theyreceived from previous task: the relationship between thegrades of the former, p(g, ti,j), and those of the latter,p(g, ti−1,j) can provide a feedback over the active criti-cal thinking component in the reputation of the membersof group g. Of course, as it is apparent that the evalua-tions over the previous product, {VAL(lp, g, ti−1,j)}p∈Ig

are coming from g’s members, for it only the teachers’grades will be taken into account.

4) finally, for each ti,j , task assigned to g, we also havethe evaluations issued by group members about theirown product p(g, ti,j): those are single learner’s self-evaluations, that can be compared with teachers’ evalu-ations, affecting learner’s self-judgment.

V. FUTURE WORK

We have presented the new SocialX system, which in-troduces collaborative group projects, and contextual micro-forums with rewards for best answers within its reputationsystem. The system is its last stages of development and willbe tested with real students during the next academic year. Ina near future, we intend to continue the SocialX expansion inseveral directions:

1) The teacher as a quality rater: Our initial approachuses the token exchanges as a simple indicator of hot topics,while the student’s judgments are used to pinpoint the mostimportant solutions to mark.

Our final aim is to transform the teacher into the ”qualityassessor” of the system, by efficiently highlighting the mostimportant didactic decisions and by leveraging the student’ssocial network structure with its reputation levels, instead thankeeping him engaged in tedious repetitive tasks.

2) Student’s Fairness: As we have seen with the discour-aged exchanges, to keep a high level of quality we discouragemisbehaviors. The penalization should be done very mildly toavoid discouraging also normal participation, thus we currentlyjust make all misbehaviors void. We would like to introducea fairness factor to capture how much the student agreeswith the ”didactic pact”, i.e. with the proper behavior rulesin the course. This factor is probably meaningful only for theteacher/tutor, and is updated whenever the student misbehaveswithin the system, either by annoying others or by trying tocheat the system.

3) Self evaluation through open-answers quizzes: We wantto introduce open-form quizzes with a very simple mechanismthat allow the student both to engage in self-evaluation andto do high-level cognitive work (respect to the Bloom’shierarchy). The student is proposed a question, which s/heanswers. Then s/he is proposed a selection of peer’s answersto the same question (including his own) from which s/hecould choose the best answer. In doing this, the student isanalyzing his and the other’s answers, comparing them toeach other. The above evaluations could be wrong, becauseit’s affected by the student competence on the topic. Thepreferences expressed in the system are then used to analyzesuch level of competence. Answers that collect high numberof choices are probably more correct, and contribute to thecompetence part of the author’s reputation. As the preferencerelation expressed should be transitive, if all preferences arecorrect they should build a poset or a total order. If a studenthas a limited knowledge of a topic and makes a wrong choicethen s/he could introduce a preference going in reverse orderrespect to the ”correct” order, which could introduce cycles.Thus, cycles in the preference graph highlight the presence ofa misunderstanding and could be used to select which answersshould be examined first to find the mistake (and to correctthe corresponding student’s competency level on that topic).Moreover, while the teacher evaluates part of the answers,the graph could be used to propagate the marks given tostudent to other (yet not examined) answers and to assess thecompetency levels of others. At a given moment, dependingon the preferences expressed so far on the presented answers,the answers can be ranked as: 1) best answers, which havebeen chosen by many; 2) worst answers, which have beenproposed but never chosen; 3) unseen answers, which haven’tbeen proposed yet. Therefore, the choice of answers to proposeto the student is better done by choosing an appropriate mixof the above three types: 1) some best answers: to allow theswitch to a better answer; 2) some worst answers: to showgood distractors; 3) some unseen answers: to evaluate all theanswers.

REFERENCES

[1] B.S. Bloom (Ed). Taxonomy of Educational Objectives. DavidMcKay Company Inc, New York (1964).

[2] Y. Cheng, H. Ku. An investigation of the effects of reciprocal peertutoring. Comp.in H.Behav. 25 (2009).

[3] G. Fernandez, A. Sterbini, M. Temperini. On the Specification ofLearning Objectives for Course Configuration. Proc. Int. Conf. onWeb-Based Education (WBE), (2007)

[4] IMS Learning Design Best Practice and Implementation Guide;IMS Learning Design Information Model; IMS Learning DesignXML Binding. http://www.imsglobal.org/learningdesign/index.cfm.

[5] A. Sterbini, M. Temperini. Learning from peers: motivating stu-dents through reputation systems. Int. Symp. on Applications andthe Internet, Social and Personal Computing for Web-SupportedLearning Communities (SPeL). Turku, Finland, (2008).

[6] A. Sterbini, M. Temperini. Adaptive Construction and Deliveryof Web-Based Learning Paths. accepted for publication in Proc.Frontiers in Education (FIE). San Antonio, Texas, (2009).

[7] E.Wenger. Communities of practice: Learning, meaning, and iden-tity. Cambridge Un. Press (1998).

[8] Yu D., Chen X. Supporting Collaborative Learning Activities withIMS LD. Proc. ICACT2007 (2007).

242242246