:Flexible Open Learner Modeling

Sergey Sosnovsky,PAWS@SIS@PITT

References

• Susan Bull, , UK.

• Mabbott, A. & Bull, S. (2004). Alternative Views on Knowledge: Presentation of Open Learner Models, ITS2004, 689-698.

• Mabbott, A. & Bull, S. (2006). Student Preferences for Editing, Persuading and Negotiating the Open Learner Model, ITS2006, 481-490.

• Kerly, A. & Bull, S. (2006). The Potential for Chatbots in Negotiated Learner Modelling, ITS2006, 443-452.

Outline

• Open User Model• Flexi-OLM:

– Viewing LM– Editing LM– Persuading LM– Negotiating LM– Multiple LM Presentations– Evaluation

• Demo

• What:Visualization of the learner model, providing a learner with a mechanism to explore it, sometimes, negotiate it.

• Why:When a learner is engaged in the analysis of the learner model he is reflecting upon his domain knowledge and experience re-calling and re-considering ideas of which he is aware.

Open Learner Modeling

Flexi-OLM• Models student understanding of basic C programming• Uses color coding for telling students about the concept

knowledge levels: – While – limited understanding– Pale yellow – somewhat limited– Yellow/green – moderate– Bright green – excellent– Red – misconception– Grey – insufficient data

• Large topics include smaller concepts. Clicking on a topic in the model brings more detailed concept-wise information about this topic understanding.

• Knowledge are assessed with the help of short questions• After playing with the system:

– Questions correspond to only one concept– No knowledge Inference between concepts– Very simple modeling formula (seems like average with linear

thresholds for knowledge levels)

Editing LM

• Flexi-OLM allows direct editing of LM

• Possible scenarios for this will be:– new learner wishes to inform

the system about topics she already understands;

– the learner grasps a concept outside the system and wants LM to reflect this;

– the learner correctly guesses a series of answers => LM has a higher knowledge level than she believes she has.

Scrutinizing and Persuading LM• Less direct method:

– A learner registers a disagreement with the LM and propose a change

– Flexi-OLM explains its believes by presenting the evidence supporting these beliefs

– If the learner still wishes to proceed, she has the opportunity to ‘persuade’ the LM by answering a series of test questions.

• Possible Scenarios:– A learner believes her

knowledge may be different than the system asserts, but lacks the confidence to edit it unchallenged,

– A learner seeks the satisfaction of proving the system wrong

Negotiating LM• Flexi-OLM supports conversation-based negotiation of LM:

– A learner is chatting with the system (as he thinks)– Flexi-OLM maintains separate believe models for LM and for a

learner– It is ensured that the same dialogue moves are available to both

parties => Each party:• has full control over their own beliefs,• can challenge the other’s belief, • can seek justification for the other’s belief (on the LM side justification

is based on the past learner’s answers),• may request justification before changing their own beliefs,• may ultimately decide to leave their belief unchanged.

– If the difference between LM’s and Student’s beliefs is:• 1 level – The LM accepts the learner’s suggestion• 2 levels – A compromise is offered

(of changing both beliefs by one level)• 3 levels – The systems seeks a justification

(the learner will be asked to answer a question)

• [possible hack] – gradual change of the LM belief by one level

Negotiating LM (cont.)The “Wizard-of-Oz” Paradigm

• Human experimenter takes the role of the chatbot – “Wizard”

• The “Wizard” follows a protocol designed to ensure:– that responses to students remained consistent between users,

and– that the ‘chatbot’ was believable to users.

• To enact the protocol, the Wizard was provided with some 350 pre-authored ‘chatbot’ negotiation initiations and responses to user inputs.

• Typical conversation:

LM Presentations in Flexi-OLM

• hierarchy, a logical grouping of related concepts;• lectures, where topics are organized the same

as in the related lecture course;• concept map showing relationships between the

topics;• prerequisites, showing possible sequences for

studying topics;• index, an alphabetical list;• ranked, where topics are listed in order of

proficiency;• textual summary.

Experiment 1 (2004)

• Two questions:– Is it beneficial for students to have a choice over

presentation of open LM, or it causes information overflow?

– Is there any strong preference for a particular LM view among individuals and if so, can it be predicted on the basis of learning style?

• 23 undergraduate students• Experiment flow:

– Pre-test (control flow in C) to populated LM– Browsing session, where students can choose among

4 different presentations

Experiment 1: Results

Experiment 2 (2006)

• Main question:– What are the students preferences concerning

editing, negotiating and persuading LM?

• 8 third-year undergraduate students• Experiment flow:

– Initial testing to populate LM– 1-hour LM exploring session (edit & persuade)– 20 minutes of negotiating with LM

Editing LM (full student

control)Negotiating LM

Persuading LM (full system

control)

Experiment 2: Results

34

11

23

36

20

Sum

Experiment 3 (2006)

• The goal:– To explore the feasibility of using a chat-based interface in an

OLM system

• 11 final year undergraduates• Experiment flow:

– Self-assessment of student knowledge of each concept, providing the initial user’s beliefs to the system.

– Interacting with the system – to populate LM and provide it with its beliefs about student knowledge

– Then students were shown how to use the chatbot and asked to interact with it for at least 20 minutes.

– Post-experiment questionnaire

Experiment 3(2006)

Demo

http://olm.eee.bham.ac.uk/flexi-olm/login.php

Thank You for Your Questions