Detecting Diligence with Online Behaviors on Intelligent...

Detecting Diligence with Online Behaviors on Intelligent Tutoring Systems

Steven DangMichael YudelsonKenneth R. Koedinger

Novice –> Expertise

2

Importance of Diligence

Ideal Reality

3

Research QuestionCan we develop a model for measuring diligence at scale using only naturalistic online behaviors?

4

The Model

Defining DiligenceWorking on academic tasks which are beneficial in the long-run but tedious in the moment, especially in comparison to more enjoyable, less effortful activities

6

Academic Diligence Task (ADT)

7

DoMath

PlayGamesor

WatchVideos

Academic Diligence Task (ADT)

8

PlayGamesor

WatchVideos

Academic Diligence Task (ADT)

9

DoMath

Personality to Behavior to Learning Outcomes

ITS

Productivity#ofproblems

Time-on-taskTimeworkingmathproblems

ADTConscientious/Self-Control

MathGrade

StandardizedTestScore

On-timeGraduation

4-yearCollegeEnrollment

DiligenceMeasures

Personality

Outcomes

10

Academic Diligence Task (ADT)

Ydiligence = f(Xtot, Xprod)

Ydiligence – Diligence Xtot – Total time workingXprod – Total problems completed

11

Diligence during online learning

12

PracticeTutor

Problems

Goofoffwith

Friends

Differences in the Task

13

AcademicDiligenceTask Adaptive OnlineLearning OldFeature

ProposedFeature

Differences in the Task

14

AcademicDiligenceTask Adaptive OnlineLearning OldFeature

ProposedFeature

timeworkedingiventimewindow

Studentsmaychoosetoworkmoreorlesstime Xtot Xtot

Differences in the Task

15

AcademicDiligenceTask Adaptive OnlineLearning OldFeature

ProposedFeature

% timeworkedinconstanttimewindow

Studentsmaychoosetoworkmoreorlesstime Xtot Xtot

Problemdifficultyislowanduniform

Varyingproblemdifficultyandadaptivenumberofexercises

Xprod Xwr

Differences in the Task

16

AcademicDiligenceTask Adaptive OnlineLearning OldFeature

ProposedFeature

% timeworkedinconstanttimewindow

Studentsmaychoosetoworkmoreorlesstime Xtot Xtot

Problemdifficultyislowanduniform

Varyingproblemdifficultyandadaptivenumberofexercises

Xprod Xwr

Cognitiveprocessisprimarilyrecallofwellknowninformation

Cognitiveprocessesinvolvemakingsenseofnewinformationandintegratingwithpriorknowledge

- Xprior

A Naturalistic Diligence Model

Ydiligence = β0Xtot + β1Xwr + β2Xprior + ε

Ydiligence – Diligence Xtot – Total time in systemXwr – Work-rate (Steps completed / Total time in system)Xprior – Prior knowledge/abilityε – Residual measurement error

17

Calculating model weights

Ydiligence ~ Ygrade

Ygrade ~ β0Xtot + β1Xwr + β2Xprior + ε

1. Calculate behavior using all student data

2. Fit the model using student outcome measure

3. Approximate YDiligence as the fitted value

18

The Dataset

Carnegie Learning CT DataWestern PA Junior High School– 426 students– 7th-9th Grades in Pre-Algebra– Full year of log data (~4M transactions)

Demographic Measures– Gender, Ethnicity, Free/Reduced Lunch

(SES)– Grade from Prior year in Math, each quarter

of study year, and End of Year Grade

20

Carnegie Learning ITS Data• Survey Measures (collected in Sept.)– Effort Regulation – Achievement Goals

• Mastery Orientation• Performance Orientation• Performance Avoidance

– Theory of Intelligence – Self-Efficacy – Interest in Math

21

Results

Convergent/Divergent Validity

Convergent/Divergent Validity

SurveyMeasure Correlation(p-value)EffortRegulation

TheoryofIntelligence

MasteryOrientation

PerformanceOrientation

PerformanceAvoidance

MathInterest

Self-Efficacy

24

(*)– Significantcorrelation (-)– Non-significantcorrelation

Convergent/Divergent Validity

SurveyMeasure Correlation(p-value)EffortRegulation (*)

TheoryofIntelligence

MasteryOrientation

PerformanceOrientation

PerformanceAvoidance

MathInterest

Self-Efficacy

25

(*)– Significantcorrelation (-)– Non-significantcorrelation

Convergent/Divergent Validity

SurveyMeasure Correlation(p-value)EffortRegulation (*)

TheoryofIntelligence (-)

MasteryOrientation (*)

PerformanceOrientation (-)

PerformanceAvoidance (-)

MathInterest (*)

Self-Efficacy (*)

26

(*)– Significantcorrelation (-)– Non-significantcorrelation

Convergent/Divergent Validity

SurveyMeasure Correlation(p-value)EffortRegulation 0.337(<.001)***

TheoryofIntelligence 0.05(.596)

MasteryOrientation .284(.003)***

PerformanceOrientation 0.189(.051)

PerformanceAvoidance .06(.52)

MathInterest 0.25(.01)**

Self-Efficacy .258(.007)**

27

Predictive Validity MethodResults

Testing Predictive Validity

29

Q1TutorUse Q2TutorUse Q3TutorUse Q4TutorUse

PriorGrade

Q1Grade

Q2Grade

Q3Grade

FinalGrade

Q4Grade

Q1Q2Q3Q4

XTot XWR XPrior YDil~

Testing Predictive Validity

Zunits = β0Ydiligence + β1Xint + β2Xdemo + β3Xprior + εZfinal_grade = β0Ydiligence + β1Xint + β2Xdemo + β3Xprior + ε

Zunits – End of year # of units completedZfinal_grade – End of year gradeYdiligence – Learned diligence parameter from modelXint – Interest in mathXdemo – Other Demographic variables (Sex, Race, SES)Xprior - Prior Year Math Grade

30

Predictive Validity (Q1Q2Q3Q4)

31

ParameterFinalGradeβ(p-value)

UnitsCompletedβ(p-value)

Diligence 1.89(<0.001)*** 1.64(<0.001)***

Predictive Validity (Q1Q2Q3Q4)

32

ParameterFinalGradeβ(p-value)

UnitsCompletedβ(p-value)

Diligence 1.89(<0.001)*** 1.64(<0.001)***PriorGrade 0.20(<0.01)**MathInterest 0.12(.073)

Predictive Validity (Q1Q2Q3Q4)

33

ParameterFinalGradeβ(p-value)

UnitsCompletedβ(p-value)

Diligence 1.89(<0.001)*** 1.64(<0.001)***PriorGrade 0.20(<0.01)** 0.07(.217)MathInterest 0.12(.073) 0.15(.017)*

Testing Predictive Validity

34

Q1TutorUse Q2TutorUse Q3TutorUse Q4TutorUse

PriorGrade

Q1Grade

Q2Grade

Q3Grade

FinalGrade

Q4Grade

Q2Q3

XTot XWR XPrior YDil~

Testing Predictive Validity

35

Q1TutorUse Q2TutorUse Q3TutorUse Q4TutorUse

PriorGrade

Q1Grade

Q2Grade

Q3Grade

FinalGrade

Q4Grade

Q2Q3

XDemo Xint XPrior YDil~

ZFG

ZUnit

Testing Predictive Validity

36

Q1TutorUse Q2TutorUse Q3TutorUse Q4TutorUse

PriorGrade

Q1Grade

Q2Grade

Q3Grade

FinalGrade

Q4Grade

Q1

XTot XWR XPrior YDil~

Testing Predictive Validity

37

Q1TutorUse Q2TutorUse Q3TutorUse Q4TutorUse

PriorGrade

Q1Grade

Q2Grade

Q3Grade

FinalGrade

Q4Grade

XDemo Xint XPrior YDil~

ZFG

ZUnit

Q1

Predictive ValidityUnitsCompleted

Samples β(p-value)

Q1Q2

Q3

Q4

Q1Q2

Q2Q3

Q3Q4

Q1Q2Q3

Q2Q3Q4

Q1Q2Q3Q4 1.64(<0.001)***

38

Predictive ValidityUnitsCompleted

Samples β(p-value)

Q1 2.23(<.001) ***Q2 1.72(<.001) ***Q3 1.16(<.001) ***Q4 2.09(<.001) ***

Q1Q2 2.17(<.001) ***Q2Q3 2.09(<.001) ***Q3Q4 1.18(<.001) ***

Q1Q2Q3 2.36(<.001) ***Q2Q3Q4 2.16(<.001)***

Q1Q2Q3Q4 1.64(<0.001)***

39

Predictive ValidityEnd-of-YearGrade

Samples β(p-value)

Q1 1.85(<.001)***Q2 1.69(<.001)***Q3 1.08(<.001) ***Q4 2.52(<.001) ***

Q1Q2 1.93(<.001) ***Q2Q3 1.95(<.001) ***Q3Q4 1.11(<.001) ***

Q1Q2Q3 2.11(<.001) ***Q2Q3Q4 2.04(<.001) ***

Q1Q2Q3Q4 1.89(<0.001)***

40

Conclusion

LimitationsHow much data is needed?– Frequency of observations– Length of sample time window

Task generalizability– Other math systems

• Alternate prior knowledge measures (eg: bkttraces)

– Other activities (eg: writing, scientific inquiry, programming)

42

Future WorkCharacterize interaction between task characteristics and diligenceInvestigating the study patterns of more/less diligent students

43

44

Matt Bernacki, Carnegie Learning, Steve Ritter, Steve Fanscali, Carnegie Mellon University, Julian Ramos Rojas, Queenie Kravitz, Scott Hudson, David Klahr, Audrey Russo, Judith Tucker, Learnlab

Datashop, Institute of Education Sciences, Program for Interdisciplinary Education Research

Acknowledgements

45

Questions?

Predictive ValidityEnd-of-YearGrade UnitsCompleted

Samples β(p-value) R2 β(p-value) R2

Q1 1.85(<.001)*** 0.40 2.23(<.001) *** 0.43

Q2 1.69(<.001)*** 0.45 1.72(<.001) *** 0.41

Q3 1.08(<.001) *** 0.45 1.16(<.001) *** 0.44

Q4 2.52(<.001) *** 0.51 2.09(<.001) *** 0.39

Q1Q2 1.93(<.001) *** 0.50 2.17(<.001) *** 0.52

Q2Q3 1.95(<.001) *** 0.56 2.09(<.001) *** 0.56

Q3Q4 1.11(<.001) *** 0.56 1.18(<.001) *** 0.55

Q1Q2Q3 2.11(<.001) *** 0.59 2.36(<.001) *** 0.63

Q2Q3Q4 2.04(<.001) *** 0.62 2.16(<.001)*** 0.62

Q1Q2Q3Q4 1.89(<0.001)*** 0.63 1.64(<0.001)*** 0.70

46

Generalizability

End-of-YearGrade

Samples β(p-value) R2

Q1 .345(.17) 0.621Q2 0.303(.43) 0.612Q3 0.450(.31) 0.615Q4 0.283(.18) 0.612

Q1Q2 0.259(.16) 0.618Q2Q3 -0.618(.54) 0.612Q3Q4 0.206(.46) 0.616

Q1Q2Q3 0.200(.26) 0.615Q1Q2Q3Q4 0.177(.30) 0.619

Repeated predictive analysis with geometry tutor

47

Predictive ValidityEnd-of-YearGrade UnitsCompleted

Samples β(p-value) R2 β(p-value) R2

Q1Q2Q3Q4 1.89(<0.001)*** 0.63 1.64(<0.001)*** 0.70Q1Q2Q3 2.11(<.001) *** 0.59 2.36(<.001) *** 0.63Q2Q3Q4 2.04(<.001) *** 0.62 2.16(<.001)*** 0.62Q1Q2 1.93(<.001) *** 0.50 2.17(<.001) *** 0.52Q2Q3 1.95(<.001) *** 0.56 2.09(<.001) *** 0.56Q3Q4 1.11(<.001) *** 0.56 1.18(<.001) *** 0.55Q1 1.85(<.001)*** 0.40 2.23(<.001) *** 0.43Q2 1.69(<.001)*** 0.45 1.72(<.001) *** 0.41Q3 1.08(<.001) *** 0.45 1.16(<.001) *** 0.44Q4 2.52(<.001) *** 0.51 2.09(<.001) *** 0.39

48