1

Reference Model for Evaluating Intelligent Tutoring Systems

Esma Aimeur, Claude Frasson

Laboratoire HERONInformatique et recherche

opérationnelleUniversité de Montréal

2Laboratoire HERON - TICE 2000

Introduction

• Goal of an ITS (Intelligent Tutoring System): produce the behaviour of an intelligent (competent) human tutor who can adapt his teaching to the learning rythm of the learner

• Ability to model and reason about domain knowledge, human thinking, learning processes, and teaching process

• building an ITS needs also to evaluate the system

• lack of evaluation methodology

3Laboratoire HERON - TICE 2000

Architecture of an ITSCurriculum

Interface

Didacticresources

Sessionmanager

Pedagogicalmodel

PlannerStudent model

Student

4Laboratoire HERON - TICE 2000

Evaluation of ITS

• Evaluation can be formative or summative

• Summative: after the design process• Formative: during the design (define

and refine goals and methods)• Evaluation techniques used in

– ITS– Software engineering

5Laboratoire HERON - TICE 2000

Evaluation of some ITS

• SHERLOCK : experimental evaluation• VCR Tutor : 4 versions compared• Formative qualitative evaluation

– iterative design– formative methods

• qualitative : characteristics of a situation• quantitative : finding causes and consequences

– case studies

6Laboratoire HERON - TICE 2000

Lester’s evaluation

• On animated pedagogical agents (Design a Plant) --- botanical and physiology

• 100 middle school students• Important educational benefits :

improved problem solving• Better than less expressive animated

agents

7Laboratoire HERON - TICE 2000

Littman- Soloway

• What is the educational impact on students ?

• What is the relationship between the architecture of an ITS and its behavior ?

• External evaluation of the student model (Proust)

• Internal evaluation : architecture, how ITS respond to input values

8Laboratoire HERON - TICE 2000

Van Lehn

• Teaching metacognitive skills to implement and evaluate an ITS (SE-Coach guide self-explanation)

• Empirical evaluation• Fundamental questions

– How the design tools are used efficiently ?– How the learner improve his knowledge ?

9Laboratoire HERON - TICE 2000

Software engineering techniques

• Product : Boehm’model – software doing what the user want it to do– use resources correctly– easy to learn– well designed, code– easily tested and maintained

10Laboratoire HERON - TICE 2000

HCI techniques

• Evaluating design– cognitive walthrough (how easy a system

is to learn)– heuristic evaluation (visibility, consistency,

flexibility, helps for the user)– model-based evaluation (GOMS) predict

user performance

11Laboratoire HERON - TICE 2000

Evaluationg Implementation

• Empirical and experimental methods• Observational methods (user

completes a set of tasks)• Query techniques

– interviews of the users about their experience

– questionnaires : questions fixed in advance

12Laboratoire HERON - TICE 2000

Learner model

• Layers : – cognitive– affective– inferential

• Learning time : speed of knowledge acquisition

• Tracability : of learner’s actions

13Laboratoire HERON - TICE 2000

Knowledge Level Explication

% of expertise

Novice No prior knowledge of the subject at all, never introduced to the subject before

0%

Beginner

Familiar with the subject. Knows some of the rules but lacks in practice, expected to answer basic questions correctly.

10-30 %

Intermediate

Learner knows most of the rules and is expected to answer correctly half of the question, while trying to perform in the other half.

40-60 %

Expert

Completely knows rules. Have ability to answer most of the questions correctly. Mainly uses the system to make his knowledge perfect.

80-100 %

Learner model

• Levels of knowledge – Based on works of

Gagné (1985)– 7 levels of knowledge– 4 principal levels

• Novice • Beginner• Intermediate• Expert

14Laboratoire HERON - TICE 2000

One-on-one

Co-learner

Learning Companion

Learning by Disturbing

Learning strategies

15Laboratoire HERON - TICE 2000

Learning strategies

• Diversity• Adaptability (switching to different

strategies)• Modification• Memorization• Feedback• Reduce cognitive load

16Laboratoire HERON - TICE 2000

Curriculum

• Generalisation• Consistency• Knowledge articulation• Reusability• Navigability• Maintenance

17Laboratoire HERON - TICE 2000

Interface

• Intuitive : learner should understand easily the functions to apply

• Interactivity : allowing the learner to be active in his learning environment

• Matching : using words, phrase and concepts

18Laboratoire HERON - TICE 2000

General

• Productivity rate : the most important factor. Time spent to produce one hour of ITS based course

• Learning outcomes : performance obtained by the learner after the session

• Ease of use of the tools

19Laboratoire HERON - TICE 2000

Conclusion

• Evaluation process is complex but needs to be realistic

• Develop first , evaluate later• Evaluation criteria as a guideline for

estimation

20Laboratoire HERON - TICE 2000

Merci de votre attention

21Laboratoire HERON - TICE 2000

Stratégies d ’apprentissageStratégies d ’apprentissage

Learner

Co-Learner

Learner TeacherCompanion

Learner Teacher

TeacherLearner

Tuteur Co-apprenant

Compagnon Apprentissage par explication