1

The Pépite project

Naima El-Kechaï, Élisabeth Delozanne, D. Prévit, B. Grugeon, F. Chenevotot

Evaluating the performance of a diagnostic system

in School Algebra

Outline Why a cognitive diagnosis ?

The Pépite project The diagnostic system

A simple example PépiDiag

Evaluation Conclusion

2

The Pépite project Objective

To design and implement a web-based application To support math teachers To manage the students’ cognitive diversity

Domain Algebra class 9th-10th grade (15-16 years old)

3

A teaching scenario Who ?

Mary is a Math teacher in 9th grade Context

To bring all her class to the same level Steps

1. Students have the online diagnostic test2. Pépite

a) analyzes the students’ answers b) displays a cognitive profile of each student and of

the whole classc) suggests 6 groups of students and a session of

exercises adapted to their cognitive profiles3. Mary can adjust the groups and the exercises4. Students log in and do the exercises in their assigned

session 4

A cognitive profile of a classGroupe A+

Engage in algebraic thinking

Use algebraic calculation appropriately

Groupe B+ Begin to use algebra to

solve problems Use sometimes mal-rules

Groupe C- Stuck in arithmetical

thinking Algebra makes no sense

5

A B+ student’s cognitive profile

6

Level Personal featuresAlgebraic calculation Techniques

Sucess rate on asked questions: 4/12Interpreting algebraic expressionsSuccess rate on asked questions: 11/23

  Strong pointsPartial mastery of algebraic rulesSome good interpretations of algebraic expressions

  Weak pointsLow mastery of algebraic calculation

Usage of algebra Mathematical modelingSuccess rate on asked questions: 5/9

  Strong pointsGood mastery of algebra usage on some problems

  Weak pointsJustification by school authority

Algebraic translation Translating situations to algebraSuccess rate on asked questions: 12/24

  Strong pointsSome good translations of mathematical relations

  Weak pointsSome translations without reformulation

More

More

More

More

More

More

A differentiated session

7

Pépite : Cognitive Diagnosis Process

8

A simple example

9

Answer assessment (local diagnosis)

10

Assessment Dimensions Values CodesValidity Incorrect V3Use of Letters Incorrect L3Translation Incorrect T332Algebraic Expressions writing

turning a product into a sum a*a-> 2a

EA41

Justification By algebra using incorrect rules J3Type of answers :

Recognition of sub-figures but turning a product into a sum

Exercise coding prescription file

Student’s answers Answers coding+ Student’s profile

PépiDiag general architecture

11

Student’s interface

PépiDiag

Teacher’s interfaceStudent

Database

Student’s answers

Exercise Database

Solves exercises

A coding prescription file (partial)Type Label Code Patterns of

equivalent expressions

1 Correct expression of the area as a product Length × Width

V1, T1,J1 (a + 3)×(b + a)

2 Correct expression of the area by adding the areas of the different rectangles

V2, T1, J1 ab+a²+3b+3a; (a×(a+b))+(3×(a+b));a(a+3)+b(a+3)

3.1 Recognition of sub-figures with parentheses errors (e.g. a+3 a+b)

V3, EA3, T3, J3 (a+3)×b+a; a+3×(b+a); a+3×b+a; a+3×a+b; a+b×a+3

3.2 Recognition of sub-figures with gathering transformations

V3, EA42, T3, J3 (a+3)(ab); (b+a)×(3a); a+3 × ab ; ab×a+3; 3a×a+b ; a+b×3a

3.3 Recognition of sub-figures but turning a product into a sum a²->2a

V3, EA41, T3, J3 5a+ab+3b; ab+2a+3b+3a

…9 No interpretation Vx 12

PépiDiag

• Local Diagnosis

• Exercise coding

prescription file

• Coding of student’s answers

• Computer• Algebra

System

• Student’ s

answers file

13

• Global Diagnosis

• Student’s profile

Outline Why a cognitive diagnosis ?

The Pépite project The diagnostic system

A simple examplePépiDiag

Evaluation Conclusion

14

Evaluation Quality requirements

1. No correct answer badly coded2. Better no code than a wrong one3. Minimal number of answers left unanalyzed

Method Comparison between experts & system

assessment 3 experts assessed 360 answers of each exercise

(one by one) Experts worked separately and then agreed on

some corrections to minor errors

15

Unanalyzed answers

16

Empty answers

Students’ answers

94/360 266/360

Data collected (on the exercise used as an example)

PépiDiag analyzes • More answers than the teacher• Less than the experienced researcher

Answers coded by PépiDiag

Answers Unanalyzed by PépiDiag

Answers Unanalyzed by 3 experts

218/266 (82%)

48/266 (18%)

24/266 (9%)

Correct answers

No correct answer is analyzed incorrectly Unanalyzed answers

5/13 : Using X instead of * 8/13 : Mixing algebra and natural language :

• « (a+b) times (3+a) » Solutions

To prevent students from typing letters other than those relating to the exercise statement

17

Correct answers

Correct automatic coding

Unanalyzed answers by PépiDiag

Unanalyzed answersBy experts

149/266 136/149 13/149 0(56%) (91%) (9 %)

Incorrect coding b+ axa +3 (parenthesis errors) b + a² + 3 (gathering the figure items) Both are incorrect (V3) distinct for human experts

and the same for the CAS Unanalyzed : incompleteness + interface

Incorrect answers

18

Incorrect answers

Correct automatic coding

Incorrect automaticcoding

Unanalyzed answers by PépiDiag

Unanalyzed answersBy experts

Unanalyzed by PépiDiagAnalyzed by experts

117/266 (44%)

80/82 2/82 35/117(30 %)

24/117(21 %)

11/117(9%)

Conclusion Our approach to asses students’ open answers

Human experts• typify anticipated patterns of correct and

incorrect solutions • code each type on several dimensions

PépiDiag, using a CAS,• matches the student’s solution with a pattern

PépiDiag is reliable 100 % : closed answers 80 % : answers with one algebraic expression 70 % : answers with a multi-step reasoning [ITS

2008] PépiDiag saves teachers’ time and tedious effort

19

Current work http://www.sesamath.net/ Diagnosis

To evaluate the building of the student’s profile from the answer assessment

Comparing• Heuristic method (implemented)• Statistical building (being implemented)• Neural Networks (being implemented)

A differentiation support tool for teachers To validate with teachers the different

learning paths adapted to each (group of profile) To build a tool to automatically

• retrieve indexed exercises from a data base • suggest a session adapted to a student’s profile 20