Collaborative Mode Complex Assessment Tasks in Authentic Context: Theoretical Background, Internet Implementation and Validation

Martin Lesage, Martin Riopel & Gilles Raîche Faculty of Education

Université du Québec à Montréal (UQÀM) Canada

lesage.martin.3@courrier.uqam.ca; riopel.martin@uqam.ca; raiche.gilles @uqam.ca

Abstract: This paper wants to develop complex assessment tasks theory by expanding the field of learning assessment and especially computer based assessment by the implementation of collaborative mode complex assessment tasks in authentic context. These tasks could include assessment modes as open questions, portfolios, scenarios, games, problems and collaborative projects. They could be performed in collaborative mode allowing many students to perform them online under teacher supervision. Furthermore, students could perform these tasks and to be assessed as team members, team leaders in training and managers in training. These assessment tasks could suppress traditional assessment techniques such as written exams in classroom and be used for certification. These tasks could be implemented using mobile learning without the presence of teachers and students with personal digital assistants(PDAs), Blackberrys and cellular phones using Internet and WIFI technology.

Introduction This paper will be a theoretical essay on collaborative mode complex assessment tasks in authentic context. It will state an implementation method for these tasks in the context of mobile learning and WIFI technology. These assessment tasks are first able to produce at regular intervals and at any time formative assessment in form of feedback to prevent failure and to provide information for further classroom intervention. They could also compute summative assessment for marking and to provide a judgment on the failure or the success of the activity. These assessment tasks could reduce the economical costs of teachers salary, traveling costs of the students and classroom allocation costs. Internet learning and assessment applications could eventually suppress teacher’s presence, classrooms and also traditional learning using assessment methods requiring presence. This paper first introduce the background and theoretical concepts of collaborative mode complex assessment tasks in authentic context. It will further suggest a user interface for an Internet based software application supporting these complex assessment tasks.

The actual research is based on the implementation of collaborative work assessment with distance learning websites. User interface models will be developed and able to implement communication between students, teachers and supervisors. The user interface could also allow a first level of assessment by enabling student and team leaders to produce reflexives practices concerning their ability to learn concepts and the mastering of abilities and skills. The second level of assessment will be performed by the team leader in training by his assessment of each team member individually and the quality of the overall work produced by his team. Every member of the team, the leader included, will be able to perform their self-assessment and to assess the work done by the other teams. Finally, the teacher or the manager in training are able to assess the overall work produced by the class or each student individually.

- 1096 -

Complex Assessment Tasks Background This section introduces theoretical concepts supporting collaborative mode complex assessment tasks in authentic context. It will define the concepts of complex assessment tasks, authentic context, collaborative mode and the related assessment and educational models.

The Rhode Island Department of Elementary and Secondary Education (RIDE, 2008) defines assessment tasks as “authentic and rigorous activities that relates to the instructional outcomes and allow students to demonstrate what they know and can do. These tasks may include oral and written responses, problem solving, journals, games and exhibitions. They determine how well students are learning, and also how well they have attained the content expectations. Assessment tasks needs to be integrated throughout the curriculum to provide on-going and comprehensive information about each student’s learning”.

The task complexity is related with knowledge assimilation and acquired skills (Louis & Bernard, 2004, p. 79; Tardif, 2006, p. 125). In certain cases, these tasks could also be performed in collaborative mode for projects realization (Kendle & Northcote, 200, p. 2). Generally, assessment tasks are asking the student to use resources and are generally utilized to measure knowledge (Scallon, 2004, p. 112). For the actual research background, a simple assessment task will be defined as an elementary method used to measure knowledge that is generally represented by multiple choice or true or false questions. The questions are used to revise course subjects and to design objective exams. Due to their trivial aspect, these questions are widely used in distance learning websites for formative and summative assessment goals.

The actual research project focuses mainly on complex assessment tasks in authentic context rather than simple assessment tasks previously described. Complex assessment tasks refers to the assessment of knowledge, abilities and skills belonging to many disciplines (Olivier, 2002). These complex assessment tasks are implemented with open questions, portfolios, practical tasks assignments, demonstrations, items, games, observations, journal, newspaper, scenarios, problems, exams and collaborative projects. (Kendle & Northcote, 2000, p. 2; RIDE, 2008).

These complex assessment tasks could be performed in an authentic context. This context is a task performing environment as close as it could be of the reality where the discipline is practiced. Louis & Bernard (2004) citing Wiggins (1993) and Hart (1993) states that contextual assessment or authentic context assessment occurs when the students are performing tasks “similar to real life situations, that are significant and creates motivation for students solving problems often found in real life context”. These complex assessment tasks in authentic context wants “the integration of acquired knowledge” and “ask the student to show is ability to apply, in a real context, knowledge, skills and attitudes that are mandatory to perform a task or solve a problem in real life” (Louis & Bernard, 2004). Durand & Chouinard (2006) adds that complex tasks context “includes the idea that an authentic learning situation implies simultaneous mobilization of various resources (knowledge, abilities, skills, attitudes, disciplinary and transversal competencies) to accomplish successful performance of a task or an activity ”. According to Fairtest (2005), these tasks “are specific activities being significant and accurate in the real world by whom students will be able to demonstrate their ideas and abilities”. The presentation mechanism of these tasks must represent the real world as accurate as possible with multimedia capabilities (audio clips, images, films, graphics animations, videos, etc.) or with virtual reality.

The proposed assessment mode is in accordance with the Québec province ministry of education (2004, p. 34) transversal competency assessment methods stating that “the assessment of transversal competencies should be based on various modalities. The assessment process could be performed by different manners: observation of the student’s behavior, mobilization and assessment of attitudes and strategies selection in learning situations by the student himself”.

The actual research project wants to improve transversal competencies assessment by the development of complex assessment tasks in collaborative mode. This collaboration mode will include four levels of assessment : team member assessment, team leader in training assessment, manager in training assessment and teacher or supervisor assessment. It will be implemented by working teams performing assessment tasks under the supervision of team managers in training and teachers. The first level of assessment will be implemented by self-assessment done by team members giving a critical regard on the accomplishment level of the tasks they are performing. The second level of assessment is implemented by the team leaders in training supervising the team members. The third level of assessment is done by the managers in training supervising the team leaders. The team leaders and managers in training can also self-assess themselves watching them manage their team and

- 1097 -

follow the realization of the team’s work. The fourth level of assessment is done by the teacher or the supervisor assessing the team members, the team leaders, the managers and the overall teamwork. Self assessment, assessment of team leaders in training and assessment of managers in training have a formative goal. Teacher’s summative assessment is made for determining the success of an activity, marking, or certification by diploma. This assessment mode can also be used into professional development where the student is replaced by workers or soldiers and the teacher by the supervisor, manager or any corporate executive. In this context, complex assessment tasks can be text composition, press reports, open questions, portfolios, exams, problems, practical projects, games or software applications, as shown in figure 1.

The students can perform these tasks alone or as team members. The requirements of the actual research project enforces team projects to obtain collaborative realization of tasks with at least two levels of supervision. According to the research project’s context, the assessment tasks will be performed with a distance assessment website.

Figure 1: Tasks products under levels of supervision.

One of the main innovations of the actual research project is to implement a multi-level collaborative assessment process in an authentic context. This context is an assessment mode that simulates real life situations. While performing an assessment task or solving a real life problem, a student could assess himself or be assessed by his peers or his teacher to be criticized. The student is then able correct his mistakes and improve his skills. In this collaborative assessment mode, formative assessment is implemented by self assessment or peers assessment. In a context of professional development, the student could also be considered as a worker. The assessment mode is similar for the worker that is assessed by his crew foreman, managers and executive directors. Assessment by these persons have often a summative assessment goal needed for employment continuation, salary increase opportunity and promotion eligibility. In contrast, the student is assessed by his team leader in training, his manager in training and by his teacher to confirm his performance success, to obtain marks, to graduate from a course or to obtain a diploma. Hence, the summative assessment is related to the certification process.

In many of these assessment situations, a student, a soldier or a worker is performing a task under the supervision of a leader, foreman, section commander or supervisor. A third supervision level could be added by an overall supervision made by a team leader, manager, teacher or executive directors that could assess teams of students or workers and also the team supervisors simultaneously as shown in figure 2.

A first example of this multi-level assessment mode could be the edition of the school’s newspapers in electronic format by students supervised by a appointed teacher. The teacher could assign some students to be journalists, columnists, clerks and programmers and other students that will be team leaders as actuality editors, column editor, managers and editors in chief. With these nominations, the teacher responsible of the school newspaper can assess individually each student appointed team member or leader by reading the edited newspaper. This example is similar as a mission performed by soldiers in the military domain: soldiers performs tasks under section commander supervision. The troop commander can assess each soldier or section commander individually. The commander can also assess the overall work done by each section or the quality of the final product by the success level of the mission.

- 1098 -

Figure 2: Complex tasks collaborative assessment mode

Collaborative mode complex assessment tasks in authentic context are based on social constructivism theories in education. Social constructivism theories states that a part of the amount of learning produced by learners are developed by teamwork and exchange of ideas between team members. Models of teaching supporting social constructivism are project based learning, cooperative learning, problem based learning and experiential learning (Raby & Viola, 2007, p. III).

Project based learning consists of the realization of a production by a team of students under the supervision of a teacher (Raby, 2007, p. 42) while cooperative learning is similar to project based learning and consists of “a kind of working and learning organization that favors interactions between peers and teamwork” (Marion, 2007, p. 70). Furthermore, problem based learning is an approach based on the grouing of students in teams to solve problem situations on which students had no prior knowledge, theory or formation. Therefore, problem situations are used to learn new knowledge (Ménard, 2007, p. 91). Finally, experiential learning is a person-centered inductive process based on prior knowledge of learners as beginning point. This process studies the learning process and its results (Ménard, 2007, p. 110).

The educational model best fitted to complex assessment tasks implementation is problem based learning because learners could be confronted to situations where they have to solve problems with no prior knowledge (Ménard, 2007, p. 91) and their realization is under the supervision of a teacher. Complex assessment task model robustness is based on the fact that the learner must be able to learn an unknown subject or performance at distance without peers or teacher supervision. Examples could be a salesman learning the functionalities of a new cellular phone or a soldier obliged to defuse an improvised explosive device to continue his mission.

Educational measurement is the ratio between a reference unit and a quantitative measure attributed to a object by an instrument (Legendre, 2005, p. 867). It is an estimation or the finding of a ratio between the size of a quantitative attribute and a unit of measurement of that size (Bertrand & Blais, 2004, p. 18). Its application includes rules of assignments of numbered quantities to objects or phenomena (Bertrand & Blais, 2004, p. 18). An educational measurement model uses a rule-based implementation. This model has two functionalities, the first is the design and implementation of measurement instruments while the second is the attribution of numbers

- 1099 -

(scores) or marks to subjects submitted to a measurement instrument. The main goal of an assessment educational model is the implementation of measurement instruments (Bertrand & Blais, 2004, p. 30). The selection of a measurement model is based on empirical methods. Experimenters first select a set of applicable measurement models, use them in the experimentation process and compare results to find the model that best fits the studied problematic (Bertrand & Blais, 2004, p. 30). There are three kinds of assessment models: classical test theory, generalizability theory and adaptive testing.

The aim of classical test theory is to verify if the mark or score obtained in a test is an accurate measurement of acquired knowledge or skills by a subject having been submitted to a test (Bertrand & Blais, 2004, p. 38). To verify if the test is accurate and well adapted to the assessment of specific skills and knowledge, the classical test theory bases itself on the following calculations and estimations : the average of measuring errors, the relationship between true scores and measuring errors, the relationship between the measurement error of two tests and the parallelism estimation of tests (Bertrand & Blais, 2004, p. 45)

Generalizability theory wants to study, analyze and differentiate measurement errors. This theory is based on generalizability models analyzing all characteristics of a measurement situation. Its aim is to identify the most important measurement errors to control them (Bertrand & Blais, 2004, p. 72). Generalizability quantification is accomplished with generalizability studies giving experimenters clues on observed scores reliability. These studies are implemented in four steps: observation, estimation, measurement and optimization (Bertrand & Blais, 2004, p. 81).

Adaptive testing is a test process where the questions, generally named items, are submitted to the subject from a question bank according to the scores attributed to the answers of the preceding questions. If the answer given by the student is wrong, the next selected item will be easier and in the opposite case, the selected item will be harder. Adaptive testing can be performed by an examiner selecting the items or by a computer selecting the item according to the item response theory (IRT) (Weiss & Kingsbury, 1984, p. 361; Stocking, 1996, p. 4; Raîche, 2000, p. 7).

These three assessment models could be applied to complex assessment tasks. Classical test theory will determine if the assessment performed by the assessment task will be accurate for the acquired knowledge and skills while generalizability theory will identify the measurement errors of the task. Finally, if the complex task includes items, their sequence of presentation will be given by adaptive testing. Complex Assessment Tasks Internet Implementation Despite that complex assessment tasks could be implemented in traditional classroom, the aim of this paper is to apply Internet and mobile technology for the implementation of assessment tasks that could be performed outside classrooms without teachers and students presence. This section will define complex assessment tasks implementation methods over the Internet, complex assessment tasks based models and multi-level team leaders in training assessment models.

Figure 3 shows two students, named “student 1” and “student N”, that are studying the notions of course X supervised by a teacher. The workstations or computers of the student 1, the student N and the teacher are connected to the Internet enabling both students to access simultaneously to the content of course X. The Internet communication software application allows the two students to communicate and work together to perform tasks or solve problems that are part of course X assessment. Another functionality of the assessment application is the real time supervising mode of the teacher having access to the students workstations. With this functionality, the teacher can see the students perform the assessments tasks related to the course while navigating through the course material using the application menus and hyperlinks. The communication application also enable the teacher to communicate in real time with the students reading course material or performing the assessment tasks to give hints, commentaries, advice or to help them in their learning process.

- 1100 -

Figure 3: Client-server architecture for assessment tasks implementation.

Competencies assessment begins by the study of course material and student attendance to courses periods or demonstrations. This attendance could be done in presence or virtually outside the classroom by Internet as studied in this paper. With knowledge acquired using distance learning applications, the student should be able to perform the task or performance required. While performing the complex assessment task, the student is able to develop reflexive practices producing self assessment of his realized work during the task. This paper studies a multi-level assessment process that includes self assessment and peers assessment types of formative assessment. These assessments provides information to the student about his knowledge, experience and capabilities to perform the task. Finally, the success of the exam, task or performance is determined by a summative assessment of the teacher regarding overall teamwork and the performances of team members, leaders and managers in training. This summative assessment could also be used to determine the success of the accomplished task for course graduation, as shown in figure 4. Individual assessment will be based on the student performance in the realization of the task and not on his teamwork capabilities. The assessment will be based on test, performances and observation. Formative assessment will be realized by self assessment, peers assessment and team leader assessment while summative assessment will be done by the teacher, as shown in figure 4. Each of these assessment will be a data entry page in the user interface.

The user interface includes screens to display the course material and screens dedicated to the realization of the assessment task able to process students data entry as text input and files. The interface will finally include a route sheet for the purpose of recording and monitoring all formative and summative assessments produced in each realization steps of the task. The teacher will use this route sheet to compute final marking or to produce a judgment on the success of the activity. The assessment functionalities of the user interface will include data entry forms to collect self assessment, team leader in training assessment, manager in training assessment and teacher summative assessment data entry, as shown in figure 7.

This paper defines theoretical notions regarding collaborative work assessment using Internet applications with assessment capabilities. User interface models will be implemented to enable communication between students and teachers for assessment purpose, as shown in figure 3. This user interface will allow a first assessment level supporting reflexive practices of students as team members, team leaders in training, managers in training and also teams as a whole concerning their abilities to acquire knowledge and skills. The second assessment level will be performed by the team leaders in training. The team leaders will be able to produce a formative assessment on each member of his team for the quality of their overall work accomplished during the task. Each team member, including the team leader, could perform their own self assessment and also produce a formative assessment of the other team members or the overall production of the class. The third assessment level will be performed by the manager in training. The manager will be able to self assess himself to produce his own opinion about his management capabilities. The manager will also be able to produce formative assessment regarding the performances of team members, team leaders, the productions of the teams and the overall production of the class. The fourth assessment level consists of the teacher’s summative assessment regarding the performances of team members, team leaders, the productions of the teams and the overall production of the class for purposes of marking, certification or to state a judgment about the success of the task, as shown in figure 5.

- 1101 -

Figure 4: Multi level assessment mode

Figure 5 is a demonstrative schema showing a competency acquirement process for students having to performs collective tasks as team members, team leaders and managers. This model enables the student to be assessed as team member, leader and manager according to a combined process of formative and summative assessment. The student will first learn individual competencies while working as a team member, as shown in the left side of figure 5. Once the student will gather enough competencies to be a productive team member, the teacher or the manager will promote him as team leader or manager to continue his training as a leader, as shown in the right side of figure 5.

Figure 5: Collective competencies acquirement process for leaders in training

- 1102 -

Figure 6 could be used for computer systems analysis of assessment software applications based on Internet. The computer analysis is used to produce system plans for teamwork complex assessment tasks software modules. These modules will assess competencies based on leadership and management capabilities.

Figure 6: Collective competencies acquirement steps The user interface will include data sheets for formative self assessment, peers assessment, team leaders assessment and managers assessments. The user interface will also provide route sheets and forms for teachers summative assessments that are the sum of all team formative and summative assessments resulting in marks, certification and judgments regarding the success of the activity, as shown in figure 7. Validation and Implementation This section will introduce approaches for the validation and implementation of collaborative mode complex assessment tasks. Validation strategies resides in the development of assessment Internet software applications.

Complex assessment tasks pertinence, validation and assessment is difficult to implement because these tasks could include many activities as performing a multiple choice test, building a current generator, conducting a military combat patrol, playing a game or showing an artistic portfolio. The assessment methods of these complex assessment tasks in generally difficult regarding the generalizability theory. Therefore, the assessment of students performances could be compared to evaluation grids or determined with quantitative measurements and qualitative observations. The assessment data could be processed with artificial intelligence applications with natural language recognition or computer vision capabilities. Tests scores and results could afterward be processed with simulation, Rash analysis and item response theory (IRT) techniques (Mislevy, 2008). Figure 7 shows a user interface screen dedicated to the computation of the teacher’s summative assessment for the overall complex task. This form will display the final marks of the student and a judgment on the success or the failure of the task. This user interface datasheet implements theory models and concepts described at the previous sections, and especially in figures 4, 5 and 6. This route sheet includes assessment parameters for team members, team leaders and managers assessment.

- 1103 -

Figure 7: Datasheet for complex assessment tasks. Conclusions

The aim of this paper was the statement of assessment tasks theoretical background, the description of assessment tasks related educational models and to analyze implementation methods of collaborative mode complex assessment tasks in authentic context. Despite complex assessment tasks could be implemented in a classroom under the supervision of a teacher, their complexity, collaborative mode and large amount of assessment data to process are strong arguments to convince teachers and course programs administrators to develop Internet software applications able to fully support them. Mobile learning technologies enables complex assessment tasks to be embedded into existing distance learning and assessment software applications. These Internet assessment applications will be able to assess student and certify courses outside classrooms without the physical presence of teachers and students. A large amount of work and research is to be done to implement these assessment Internet websites and especially in the case of mobile technologies applications. The actual research will expand the fields of artificial intelligence, user interface development, tree data structures and web server database XML programming. References Bertrand, R. & Blais, J. G. (2004). Modèles de mesure – L’apport de la théorie des réponses aux items. Sainte-

Foy, Québec : Presses de l’Université du Québec. Durand, M. J. & Chouinard, R. (2006). L’évaluation des apprentissages: De la planification de la démarche à

la communication des résultats. Montréal : Éditions Hurtubise HMH ltée. Fairtest. (2005). Queensland, Australia, “ Rich Tasks Assessment Program ” [On line].

Access:http://www.fairtest.org/queensland-australia-rich-tasks-assessment-program Fraley, C. & Raftery, A. E. (1998). How Many Clusters? Which Clustering Method? Answers Via Model-Based

Cluster Analysis. The Computer Journal 41(8), 578-588. Illingworth,V. (Ed.). (1996). Dictionary of computing. New York: Oxford University Press. IMS Global Learning Consortium, Inc (2007). QTI standard [On line]. Access:

http://www.imsglobal.org/question/ Kendle, A. & Northcote, M. (2000). The Struggle for Balance in the Use of Quantitative and Qualitative Online

Assessment Tasks. Proceedings ASCILITE 2000.

- 1104 -

Ketchen, D. J. Jr. & Shook, C. L. (1996). The application of cluster analysis in strategic management research: An analysis and critique. Strategic Management Journal, 17, 441-458.

Legendre, R. (2005). Dictionnaire actuel de l’éducation (3è ed.). Montréal : Guérin. Le petit Larousse Illustré (2008). Paris : Les éditions Larousse. Louis, R. & Bernard, H. (2004). L’évaluation des apprentissages en classe. Montréal : Groupe Beauchemin

éditeur ltée. Marion, I. (2007). Apprentissage coopératif. In C. Raby & S. Viola (Eds.), Modèles d’enseignement et théories

d’apprentissage : De la pratique à la théorie (p. 67-89). Montréal: Les Éditions CEC Inc. Ménard, L. (2007). Apprentissage par problèmes. In C. Raby & S. Viola (Eds.), Modèles d’enseignement et

théories d’apprentissage : De la pratique à la théorie (p. 90-106). Montréal: Les Éditions CEC Inc. Ménard, L. (2007). Apprentissage expérientiel. In C. Raby & S. Viola (Eds.), Modèles d’enseignement et

théories d’apprentissage : De la pratique à la théorie (p. 107-118). Montréal: Les Éditions CEC Inc. Ministère de l’Éducation, du loisir et du sport, Gouvernement du Québec (MELS). (2005). L’évaluation des

apprentissages au secondaire : Cadre de référence, Ministère de l’Éducation, du loisir et du sport, Direction générale de la formation des jeunes [On line]. Access : http://www.mels.gouv.qc.ca/sections/publications/publications/EPEPS/Formation_jeunes/Evaluation/1.pdf

Ministère de l’Éducation, du loisir et du sport, Gouvernement du Québec (MELS). (2004). Programme de formation de l'école québécoise : Enseignement secondaire, premier cycle, Ministère de l’Éducation, du loisir et du sport, Direction générale de la formation des jeunes [On line]. Access : http://www.mels.gouv.qc.ca/DFGJ/dp/programme_de_formation/secondaire/pdf/prform2004/chapitre003v2.pdf

Mislevy, R.J. (2008). How cognitive science challenges the educational measurement tradition. Measurement: Interdisciplinary Research and Perspectives, 6, 124.

Olivier, L. (2002). L’évaluation des apprentissages [On line]. Commission Scolaire de Montréal. Access: http://www.csdm.qc.ca/Csdm/Administration/pdf/reforme_vol1_no2.pdf

Raby, C. (2007). L’apprentissage par projets. In C. Raby & S. Viola (Eds.), Modèles d’enseignement et théories d’apprentissage : De la pratique à la théorie (p. 40-66). Montréal : Les Éditions CEC inc.

Raby, C. & Viola, S. (2007). Modèles d’enseignement et théories d’apprentissage : De la pratique à la théorie. In C. Raby & S. Viola (Eds.), Modèles d’enseignement et théories d’apprentissage : De la pratique à la théorie. Montréal : Les Éditions CEC inc.

Raîche, G. (2000). La distribution d’échantillonnage de l’estimateur du niveau d’habileté en testing adaptatif en fonction de deux règles d’arrêt : selon l’erreur type et selon le nombre d’items administrés. Doctoral thesis, Université de Montréal.

Raîche, G. (2004). Le testing adaptatif. In R. Bertrand & J.-G. Blais (Eds.), Modèles de mesure : l'apport de la théorie des réponses aux items. Montréal: Presses de l'université du Québec.

RIDE (2008). Statewide Curriculum [On line]. Access : http://www.ride.ri.gov/Instruction/curriculum/rhodeisland/assessment/tasks.htm

Scallon, G. (2004) L’évaluation des apprentissages dans une approche par compétences. Montréal: Éditions du Renouveau Pédagogique.

Stocking, M. L. (1996). Revising Answers to Items in Computerized Adaptive Tests : A Comparison of Three Models (Report No: TES-RR-96-12). Princeton, N.J: Educational Testing Service.

Tardif, J. (2006). L’évaluation des compétences. Documenter le parcours de développement. Montréal : Chenelière Éducation.

Weiss, D. J. (1983). Introduction. In D. J. Weiss (Eds.) : New horizons in testing - Latent trait test theory and computerized adaptive testing (p. 1-7). New York : Academic Press.

Weiss, D. J. & Kingsbury, G. G. (1990). Application of computerized adaptive testing to educational problems. Journal of Educational Measurment, 21(4), 361-375.

XML (2007). Extensible Markup Language. [En ligne] Accès: http://www.w3.org/XML/

- 1105 -