Assessment embedded in step-based tutors (SBTs)

CPI 494 Feb 12, 2009

Kurt VanLehn

ASU

Outline

Benefits of assessment embedded in SBT Synthesis of several common ITS assessment

methods– Terminology– Where does assessment fit into an ITS?– A synthesis

Why should step-based tutors (SBT) be better than multiple choice tests (MCT) for assessment?

Why is a 30 minute task (SBT) better than a 1 minute task (MCT)?– Authenticity, validity, prediction of valued tasks– “teaching to the test” becomes valuable

Why is seeing the steps better than seeing just the answer?– Assess problem solving strategy– Assess meta-cognitive strategy (e.g., use of hints)

Why is embedding assessment in daily homework or seatwork better than monthly in-class testing?

Frees up time for teaching Can’t be absent on day of test “doesn’t test well” and “only good at tests” are

no longer excuses No test anxiety No cramming Instructors can see changes in competence daily

Homework is unsupervised but testing is supervised. How does that affect assessment?

What if students refer to textbook, web, friends or instructor?– Sequestered problem solving is inauthentic– Assessment may improve if tutor can “see” such

references What about cheating?

– Technology can thwart most copying– Would have to hire a friend to do almost all of one’s

homework

Outline

Benefits of embedding assessments in SBTs Synthesis of several common ITS assessment

methods– Terminology– Where does assessment fit into an ITS?– A synthesis

Next

Definition: Knowledge component

A piece of knowledge Generic – can be applied in to multiple tasks Any form: rule, principle, concept, fact,

procedure… Examples:

– Two angles are called “supplementary” if….– To buckle your seat belt, insert the …– The gravitational constant of earth, g, is 9.8 m/s2.

Definition: Mastery of a knowledge component (KC)

No more training needed– E.g., The ITS need no longer pick problems that

involve it Typical signs of mastery

– Applies the KC when should– Applies the KC only when should– Applies without referring to text, notes, …– Can explain the KC, when to apply, etc.

Main goal of assessment

Never have enough data – E.g., typically do not have student’s explanations

Must estimate P(mastery) for every knowledge componentgiven only steps as data

Definition: The response pattern of a student’s step

How a step was accomplished by the student E.g., 4 response patterns:

(S = student; T = tutoring system)

1. S: Correct.

2. S: Error; T: Hint; S: Correct.

3. S: Help; T: Hint; S: Correct.

4. S: Help; T: Hint; S: Help; T: Hint; S: Help; T: Bottom out hint; S: Correct

Definition: The relevant knowledge components of a student’s step

The knowledge components involved in deriving the correct step– given the information available at the time the step

was done The knowledge the student should apply to

generate this step The topic of the hints (if any were given)

Outline

An introduction to a common type of ITS– Terminology– Three examples

Synthesis of several common ITS assessment methods– Terminology (KC, mastery, response pattern, KCs

relevant to a step)– Where does assessment fit into an ITS?– A synthesis

Next

The 3 major computations of intelligent tutoring systems

P(mastery) for each knowledge component

Problem to be solved

All correct steps in all orders

Response pattern for each student step

Expert

Assessor

Helper

The expert’s computation Expert can be humans

or an expert system Solve the problem

in all acceptable ways Record steps taken Record knowledge

components used at each step

P(mastery) for each knowledge component

Problem to be solved

All correct steps in all orders

Response patterns for each student step

Expert

Assessor

Helper

The helper’s computation

When the student enters a step, match it to a correct step

Give feedback & hints as necessary

Record response pattern

P(mastery) for each knowledge component

Problem to be solved

All correct steps in all orders

Response patterns for each student step

Expert

Assessor

Helper

The assessor’s computation Given

– Reponse patterns for each step taken by the student

– Old P(mastery) for each knowledge component

Calculate– New P(mastery)

P(mastery) for each knowledge component

Problem to be solved

All correct steps in all orders

Response patterns for each student step

Expert

Assessor

Helper

Outline

An introduction to a common type of ITS– Terminology– Three examples

Synthesis of several common ITS assessment methods– Terminology– Where does assessment fit into an ITS?– A synthesis Next

Some assessment methodsKnowledge tracing (Corbett & Anderson)-- Only two response patterns: “S:Correct” & “other”-- Only one relevant knowledge component per step

SMART (Shute)-- Multiple response patterns

Andes (VanLehn et al.)-- Multiple knowledge components per step

Today: A synthesis of the above

DT tutor (Murray)-- Dynamic Bayesian networks

The problem: Given step responses & prior P(mastery), calculate posterior P(mastery)

KC1

KC3

KCn

KC2

KC4

KC1

KC3

KCn

KC2

KC4

Step 1

S: error; T: Hint;

S: Correct

Step 2

S: Correct

Step 3

S: Help;T: Hint;S: Help;T: Hint;S: Help;

T: B.O. Hint;S: Correct

.05

.05

.11

.10

.05

.75

.12

.50

.99

.05

Prior P(mastery) for each knowledge component (KC)

Posterior P(mastery) for each knowledge component

A dynamic Bayesian network represents that P(mastery) changes during steps

Bayesiannetwork

KC1

KC3

KCn

KC2

KC4

KC1

KC3

KCn

KC2

KC4

KC1

KC3

KCn

KC2

KC4

Bayesian network

Step 1 Step 2

P(mastery) of knowledge component 4 at time 1

P(mastery) of knowledge component 4 at time 2

Assume P(mastery) changes only for relevant knowledge components

Bayesian network

KC1

KC3

KCn

KC2

KC4

KC1

KC3

KCn

KC2

KC4

KC1

KC3

KCn

KC2

KC4Bayesian network

Step 1Involves only KC2 and KC3

Step 2Involves only KC1 and KC4

Knowledge tracing: Assumes just one relevant knowledge component per step

Bayesian network

KC1

KC3

KCn

KC2

KC4

KC1

KC3

KCn

KC2

KC4

KC1

KC3

KCn

KC2

KC4 Bayesian network

Step 1Involves only

KC2

Step 2Involves only

KC4

Knowledge tracing: Network topology and conditional probability tables

KC1

KC3

KCn

KC2

KC4

KC1

KC3

KCn

KC2

KC4

KC1

KC3

KCn

KC2

KC4

Step 1 Step 2

Step response

Step responseKC2 mastered: Yes No

S: Correct .92 .10

Other .08 .90

Was mastered: Yes No

Mastered 1 .47

Not mastered 0 .53

P(Step response not correct | KC2 mastered)

Called “slip parameter”

P(Step response correct | KC2 not mastered)

Called “guess parameter”

P(Mastered | Was not mastered, relevant to step) Called “acquisition parameter”

Need to allow multiple relevant knowledge components per step

KC1

KC3

KCn

KC2

KC4

Step response

KC1

KC3

KCn

KC2

KC4

KC1

KC3

KCn

KC2

KC4

Step 1 Step 2

Step response

Use AND node to represent that the step response depends only on

whether or not all relevant knowledge components are mastered

The student’s meta-cognitive strategy also affects the student’s step response

KC1

KC3

KCn

KC2

KC4

Step response

KC1

KC3

KCn

KC2

KC4

KC1

KC3

KCn

KC2

KC4

Step 1 Step 2

Step response

Meta-cognitive strategy

Conditional probability table for the first “Step response” node

Meta-cognitive strategy Learn Perform

Both KC2 and KC3 mastered? Yes No Yes No

S: Correct .99 .50 .99 .10

S: Error; T: Hint; S: Correct .01 .35 .00 .01

S: Help;… T: Bottom out hint; S: Correct .00 .01 .01 .85

All other response patterns .00 .14 .00 .04

P(step response was “S:Correct” | Meta-cognitive strategy was “Learn”,

Not all knowledge components mastered before step

Strong tendency to abuse help when the meta-cognitive strategy is “Perform” and some relevant knowledge component is not mastered yet.

Need to assume new P(mastery) depends on the step response as well as the old P(mastery)

KC1

KC3

KCn

KC2

KC4

Step response

KC1

KC3

KCn

KC2

KC4

KC1

KC3

KCn

KC2

KC4

Step 1 Step 2

Step response

Meta-cognitive strategy

Conditional probability table for a “knowledge component” node

Step response: S:Correct 1 hint Bottom out hint Other

Mastered before? Yes No Yes No Yes No Yes No

Mastered after step 1.0 .50 .95 .30 .90 .01 .55 .05

Not mastered after step 0.0 .50 .05 .70 .10 .99 .45 .95

How the network should be used

Given log file – all steps taken by the student Construct the whole network – all steps in all problems Enter prior P(mastery) on knowledge components at

time 1 Clamp all step responses Update – a huge computation Read out P(mastery) on knowledge components after

last step.

How the network is actually used Step 1

– Construct network– Enter prior P(mastery)– Clamp step 1 response– Update– Read out P(mastery)

Step 2– Construct network– Enter prior P(mastery)– Clamp step 2 response– Update– Etc.

KC1

KC3

KCn

KC2

KC4

Step response

KC1

KC3

KCn

KC2

KC4

Step 1

Meta-cognitive strategy

KC1

KC3

KCn

KC2

KC4

KC1

KC3

KCn

KC2

KC4

Step 2

Step response

Meta-cognitive strategy

Future workKnowledge

tracingSMART Andes Today

Design Implementation Calibration Validity Sensitivity

Summary

An introduction to a common type of ITS– Steps replace answers as main student response

A synthesis of several Bayesian assessment methods– Many knowledge components– Probability of mastery of each knowledge component– One (or more) knowledge components per step– Can use step response patterns to infer mastery– Global traits e.g., meta-cognitive strategy