What Should I Do Next? Adaptive Sequencing in the Context of Open Social Student Modeling

Roya Hosseini, I-Han Hsiao, Julio Guerra, Peter Brusilovsky

PAWS Lab University of Pittsburgh

Overview

• Motivation – why do we care about guidance?

• Past work – how to guide students to the right content?

• Current work – adaptive sequencing combined with social guidance – what we learned from the classroom study

• Work in progress & future work

2

Motivation

Goal – personalized guidance to the most appropriate educational

content for each learner

!

Why personalized guidance? – helps students acquire knowledge faster – improves learning outcomes – reduces navigational overhead – increases student motivation to work with content

3

Existing Guidance Technologies

1. Knowledge-based approaches • decide the most appropriate content for an individual with

respect to the domain model, student model, and course goal • adaptation type:

• fine-grained concept-based (ELM-ART, NavEx) • coarse-grained topic-based (QuizGuide) !

2. Social guidance4

Concept-Based Adaptation

Example 2 Example M

Example 1

Problem 1

Problem 2 Problem K

Concept 1

Concept 2

Concept 3

Concept 4

Concept 5

Concept N

Examples

Problems

Concepts

5

ELM-ART: Adaptive Link Annotation in LISP

6

green bullet indicates a recommended page

red bullet indicates a page user is not ready for

G. Weber And P. Brusilovsky, IJAIED 2001. Elm-Art: An Adaptive Versatile System For Web-Based Instruction

NavEx: Concept-Based Adaptive Navigation Support

bullet is filled based on progress

font style denotes the relevance of example

a relevant example with no progress

an example not ready to be browsed

7

M. Yudelson And P. Brusilovsky, AIED 2005. Navex: Providing Navigation Support For Adaptive Browsing Of Annotated Code Examples.

Topic-Based Adaptation

• each topic is associated with a number of educational activities !

• each activity is classified under 1 topic

8

Topic

A Topic

B

Topic

C

QuizGuide : Topic-Based Adaptive Navigation Support

Current quiz

number of arrows: knowledge in the topic (0-3)

color Intensity: learning goal

P. Brusilovsky, S. Sosnovsky And O. Shcherbinina, E-Learn 2004. Quizguide: Increasing The Educational Value Of Individualized Self-Assessment Quizzes With Adaptive Navigation Support. 9

curre

ntpre

requis

iteno

t-rele

vant

not-r

eady

Knowledge Maximizer Paradigm

10Hosseini, R., Brusilovsky, P., & Guerra, J. (AIED 2013, January). Knowledge Maximizer: Concept-based Adaptive Problem Sequencing for Exam Preparation.

Learn maximum knowledge from next activity while controlling prerequisites

Existing Guidance Technologies

1. Knowledge-based approaches 2. Social guidance

• uses Open Social Student Modeling (OSSM) • students can view each others’ or class knowledge model • almost as efficient as knowledge-based guidance

- higher success rates & engagement - much less knowledge engineering overhead

• drawback: make students more conservative in their work !! 11

Mastery Grids: Topic-based Navigation Support in OSSM Platform

anonymized ranked list of peers and their topic-based progress

position of current student in class

topic-based progress of student

topic-based progress of class

Loboda, T. D., Guerra, J., Hosseini, R., & Brusilovsky, P. (EC-TEL 2014). Mastery Grids: An Open Source Social Educational Progress Visualization. 12

• combines social guidance with knowledge-based guidance

• enhances the approach to maximize student knowledge

• implements the guidance in context of Mastery Grids OSSM

• reports the results from the classroom study

Sequencing + Open Social Student Modeling

13

Greedy Sequencing (GS)

• aims at maximizing student knowledge in domain concepts • concept-based adaptation:

- uses prerequisite and outcome concepts in content items

14

User%Modeling%database%

Greedy%Sequencing%

Knowledge%Report%Service%

Rank%C1%

Prerequisites%Outcomes%

Content%C1:%Concepts%

Greedy Sequencing: Content Ranking by Knowledge Maximization

15

amount of known prerequisites

amount of unknown outcomes

rank of the content, [0-1]

number of outcomes

np:number of prerequisites ki: knowledge of concept i wi: weight of concept i, log(tf-idf value)

16

• marked top three recommendations generated by GS • size of star shows relative rank of content

- bigger star —> higher priority

The Study

143 undergraduates in ASU (Fall 2014) Java Programming & Data Structure course ‣ 111 problems — 103 examples — 19 topics

!Study had 2 main Parts (1) no sequencing (Aug. 21 – Sep. 25) (2) with sequencing (Sep. 26 – Oct. 21) • 86 subjects logged into the system • we considered 53 subjects with problem attempts >= 30

17

Navigational Pattern Analysis

GS breaks out the common path of social guidance

0.08

0.08

0.16

0.68

0.06

0.05

0.12

0.78

0.17

0.17

0.2

0.47

Jump−Backward

Jump−Forward

Next−Topic

Within−Topic

Part 1 Part 2−N Part 2−R

when following GS, “groupthink” stay on the current topic shortens considerably !students moved to next topic more quickly & expanded their non-sequential navigation

Value of GS on Amount of Learning & Speed

19

Learning gain: • no significant differences in the learning gain

- non-followers (M = 0.50, SD = 0.27) - followers (M=0.44, SD=0.23) !

Learning speed: (learning gain/number of problem attempts)×100

! • speed of learning was higher among the followers - non-followers (M = 0.54%, SD = 0.27%) - followers (M = 0.97%,SD = 0.88%) speed increased about twice - p = .083, using a Welch t-test

Value of GS on Learning & Speed: Weak vs. Strong Students

20

0.00#

0.20#

0.40#

0.60#

0.80#

1.00#

1.20#

1.40#

1.60#

1.80#

2.00#

Weak#students# Strong#students#

%#Learning#speed##

Non;followers# Followers#

0"

0.1"

0.2"

0.3"

0.4"

0.5"

0.6"

0.7"

0.8"

0.9"

Weak"students" Strong"students"

Normalize

d"learning"gain"

Non?followers" Followers"

• no significant differences in learning gain • followers with high prior knowledge learn faster (p=.039)

Value of GS on Problem Solving Performance

21

Correctness is more frequent in recommended problems • odds of correct answer in a problem offered by GS was 1.59

(SE = 0.19) times more than a not-recommended problem

How: • data collected from part 1 and 2 of study (5760 problem attempts: 5275 not-recommended, 485 offered by GS) • fitted a logistic mixed effects model • fixed effect: attempt type (recommended, not-recommended) • response variable: correctness of attempt (0/1)

Value of GS on Class Performance

22

An attempt on a GS recommendation was associated with higher grade

• attempting a recommended content (problem/example) was associated with 0.56 increase in final grade (SE=0.24, p=.017)

~ 9 times greater than the effect of a not-recommended content

How: • data of 40 students (had exam score + used system) • fitted regression model to predict exam grade using number of attempts on contents

• 6 questions (5-point Likert scale) • data collected from 51 students (answered questionnaire + used the system)

M:4.1 M:3.9 M:3.1 M:3.8 M:4.2M:2.4

Subjective Feedback

23

like

star

usefu

lcle

ar ! re

ason

distra

ctive

Wrap Up

adaptive sequencing + social guidance: !✓encouraged non-sequential navigation patterns  ✓increased learning speed of stronger students

‣ more optimal content navigation ✓was positively related to student performance

‣ higher exam score ‣ more success in problems

Work in Progress & Future Work

๏ running study with over 200 students in ASU - GS vs. probabilistic approach based on FAST

!๏ what is the best way to visualize student/class data?

- alternatives to topic-based guidance (2D content maps )

!๏ how to increase students’ awareness of recommendations?

- adding annotations, …

ReferencesKnowledge Maximizer: Hosseini, R., Brusilovsky, P., & Guerra, J. (2013, January). Knowledge Maximizer: Concept-based Adaptive Problem Sequencing for Exam Preparation. In Artificial Intelligence in Education (pp. 848-851). Springer Berlin Heidelberg.!Mastery Grids: Loboda, T. D., Guerra, J., Hosseini, R., & Brusilovsky, P. (2014). Mastery Grids: An Open Source Social Educational Progress Visualization. In Open Learning and Teaching in Educational Communities (pp. 235-248). Springer International Publishing. !QuizGuide: P. Brusilovsky, S. Sosnovsky And O. Shcherbinina, 2004. Quizguide: Increasing The Educational Value Of Individualized Self-Assessment Quizzes With Adaptive Navigation Support. In: J. Nall And R. Robson, Eds., World Conference On Elearning, E-Learn 2004 Aace, Washington, Dc, Usa, 1806-1813.!NavEx: M. Yudelson And P. Brusilovsky, 2005. Navex: Providing Navigation Support ForAdaptive Browsing Of Annotated Code Examples. In: C.-K. Looi, G. Mccalla, B. Bredeweg And J. Breuker, Eds., 12Th International Conference On Artificial Intelligence In Education, Ai-Ed'2005 Ios Press, Amsterdam, The Netherlands, 710-717.!ELM-ART: G. Weber And P. Brusilovsky, 2001. Elm-Art: An Adaptive Versatile System For Web-Based Instruction. International Journal Of Artificial Intelligence In Education, 12 (4), 351-384

26

Thank You!

Intelligent Systems Program

Roya Hosseini roh38@pitt.edu

Peter Brusilovsky peterb@pitt.edu

I-Han (Sharon) Hsiao sharon.hsiao@asu.edu

Julio Guerra jdg60@pitt.edu

Try it! adapt2.sis.pitt.edu/kt/mg-gs.html

https://www.youtube.com/watch?v=Kak8F2y5GkU