Detecting Diligence with Online Behaviors on Intelligent ...learningatscale.acm.org/las2017/wp-content/uploads/... · Online Behaviors on Intelligent Tutoring Systems Steven Dang

Detecting Diligence with Online Behaviors on Intelligent Tutoring Systems

Steven DangMichael YudelsonKenneth R. Koedinger

Novice –> Expertise

2

Importance of Diligence

Ideal Reality

3

Research QuestionCan we develop a model for measuring diligence at scale using only naturalistic online behaviors?

4

The Model

Defining DiligenceWorking on academic tasks which are beneficial in the long-run but tedious in the moment, especially in comparison to more enjoyable, less effortful activities

6

Academic Diligence Task (ADT)

7

DoMath

PlayGamesor

WatchVideos


8

PlayGamesor

WatchVideos


9

DoMath

Personality to Behavior to Learning Outcomes

ITS

Productivity#ofproblems

Time-on-taskTimeworkingmathproblems

ADTConscientious/Self-Control

MathGrade

StandardizedTestScore

On-timeGraduation

4-yearCollegeEnrollment

DiligenceMeasures

Personality

Outcomes

10


Ydiligence = f(Xtot, Xprod)

Ydiligence – Diligence Xtot – Total time workingXprod – Total problems completed

11

Diligence during online learning

12

PracticeTutor

Problems

Goofoffwith

Friends

Differences in the Task

13

AcademicDiligenceTask Adaptive OnlineLearning OldFeature

ProposedFeature


14


ProposedFeature

timeworkedingiventimewindow

Studentsmaychoosetoworkmoreorlesstime Xtot Xtot


15


ProposedFeature

% timeworkedinconstanttimewindow


Problemdifficultyislowanduniform

Varyingproblemdifficultyandadaptivenumberofexercises

Xprod Xwr


16


ProposedFeature

% timeworkedinconstanttimewindow


Problemdifficultyislowanduniform

Varyingproblemdifficultyandadaptivenumberofexercises

Xprod Xwr

Cognitiveprocessisprimarilyrecallofwellknowninformation

Cognitiveprocessesinvolvemakingsenseofnewinformationandintegratingwithpriorknowledge

- Xprior

A Naturalistic Diligence Model

Ydiligence = β0Xtot + β1Xwr + β2Xprior + ε

Ydiligence – Diligence Xtot – Total time in systemXwr – Work-rate (Steps completed / Total time in system)Xprior – Prior knowledge/abilityε – Residual measurement error

17

Calculating model weights

Ydiligence ~ Ygrade

Ygrade ~ β0Xtot + β1Xwr + β2Xprior + ε

1. Calculate behavior using all student data

2. Fit the model using student outcome measure

3. Approximate YDiligence as the fitted value

18

The Dataset

Carnegie Learning CT DataWestern PA Junior High School– 426 students– 7th-9th Grades in Pre-Algebra– Full year of log data (~4M transactions)

Demographic Measures– Gender, Ethnicity, Free/Reduced Lunch

(SES)– Grade from Prior year in Math, each quarter

of study year, and End of Year Grade

20

Carnegie Learning ITS Data• Survey Measures (collected in Sept.)– Effort Regulation – Achievement Goals

• Mastery Orientation• Performance Orientation• Performance Avoidance

– Theory of Intelligence – Self-Efficacy – Interest in Math

21

Results

Convergent/Divergent Validity


SurveyMeasure Correlation(p-value)EffortRegulation

TheoryofIntelligence

MasteryOrientation

PerformanceOrientation

PerformanceAvoidance

MathInterest

Self-Efficacy

24

(*)– Significantcorrelation (-)– Non-significantcorrelation


SurveyMeasure Correlation(p-value)EffortRegulation (*)

TheoryofIntelligence

MasteryOrientation

PerformanceOrientation

PerformanceAvoidance

MathInterest

Self-Efficacy

25



SurveyMeasure Correlation(p-value)EffortRegulation (*)

TheoryofIntelligence (-)

MasteryOrientation (*)

PerformanceOrientation (-)

PerformanceAvoidance (-)

MathInterest (*)

Self-Efficacy (*)

26



SurveyMeasure Correlation(p-value)EffortRegulation 0.337(<.001)***

TheoryofIntelligence 0.05(.596)

MasteryOrientation .284(.003)***

PerformanceOrientation 0.189(.051)

PerformanceAvoidance .06(.52)

MathInterest 0.25(.01)**

Self-Efficacy .258(.007)**

27

Predictive Validity MethodResults

Testing Predictive Validity

29

Q1TutorUse Q2TutorUse Q3TutorUse Q4TutorUse

PriorGrade

Q1Grade

Q2Grade

Q3Grade

FinalGrade

Q4Grade

Q1Q2Q3Q4

XTot XWR XPrior YDil~


Zunits = β0Ydiligence + β1Xint + β2Xdemo + β3Xprior + εZfinal_grade = β0Ydiligence + β1Xint + β2Xdemo + β3Xprior + ε

Zunits – End of year # of units completedZfinal_grade – End of year gradeYdiligence – Learned diligence parameter from modelXint – Interest in mathXdemo – Other Demographic variables (Sex, Race, SES)Xprior - Prior Year Math Grade

30

Predictive Validity (Q1Q2Q3Q4)

31

ParameterFinalGradeβ(p-value)

UnitsCompletedβ(p-value)

Diligence 1.89(<0.001)*** 1.64(<0.001)***


32



Diligence 1.89(<0.001)*** 1.64(<0.001)***PriorGrade 0.20(<0.01)**MathInterest 0.12(.073)


33



Diligence 1.89(<0.001)*** 1.64(<0.001)***PriorGrade 0.20(<0.01)** 0.07(.217)MathInterest 0.12(.073) 0.15(.017)*


34


PriorGrade

Q1Grade

Q2Grade

Q3Grade

FinalGrade

Q4Grade

Q2Q3



35


PriorGrade

Q1Grade

Q2Grade

Q3Grade

FinalGrade

Q4Grade

Q2Q3

XDemo Xint XPrior YDil~

ZFG

ZUnit


36


PriorGrade

Q1Grade

Q2Grade

Q3Grade

FinalGrade

Q4Grade

Q1



37


PriorGrade

Q1Grade

Q2Grade

Q3Grade

FinalGrade

Q4Grade

XDemo Xint XPrior YDil~

ZFG

ZUnit

Q1

Predictive ValidityUnitsCompleted

Samples β(p-value)

Q1Q2

Q3

Q4

Q1Q2

Q2Q3

Q3Q4

Q1Q2Q3

Q2Q3Q4

Q1Q2Q3Q4 1.64(<0.001)***

38

Predictive ValidityUnitsCompleted

Samples β(p-value)

Q1 2.23(<.001) ***Q2 1.72(<.001) ***Q3 1.16(<.001) ***Q4 2.09(<.001) ***

Q1Q2 2.17(<.001) ***Q2Q3 2.09(<.001) ***Q3Q4 1.18(<.001) ***

Q1Q2Q3 2.36(<.001) ***Q2Q3Q4 2.16(<.001)***

Q1Q2Q3Q4 1.64(<0.001)***

39

Predictive ValidityEnd-of-YearGrade

Samples β(p-value)

Q1 1.85(<.001)***Q2 1.69(<.001)***Q3 1.08(<.001) ***Q4 2.52(<.001) ***

Q1Q2 1.93(<.001) ***Q2Q3 1.95(<.001) ***Q3Q4 1.11(<.001) ***

Q1Q2Q3 2.11(<.001) ***Q2Q3Q4 2.04(<.001) ***

Q1Q2Q3Q4 1.89(<0.001)***

40

Conclusion

LimitationsHow much data is needed?– Frequency of observations– Length of sample time window

Task generalizability– Other math systems

• Alternate prior knowledge measures (eg: bkttraces)

– Other activities (eg: writing, scientific inquiry, programming)

42

Future WorkCharacterize interaction between task characteristics and diligenceInvestigating the study patterns of more/less diligent students

43

44

Matt Bernacki, Carnegie Learning, Steve Ritter, Steve Fanscali, Carnegie Mellon University, Julian Ramos Rojas, Queenie Kravitz, Scott Hudson, David Klahr, Audrey Russo, Judith Tucker, Learnlab

Datashop, Institute of Education Sciences, Program for Interdisciplinary Education Research

Acknowledgements

45

Questions?

Predictive ValidityEnd-of-YearGrade UnitsCompleted

Samples β(p-value) R2 β(p-value) R2

Q1 1.85(<.001)*** 0.40 2.23(<.001) *** 0.43

Q2 1.69(<.001)*** 0.45 1.72(<.001) *** 0.41

Q3 1.08(<.001) *** 0.45 1.16(<.001) *** 0.44

Q4 2.52(<.001) *** 0.51 2.09(<.001) *** 0.39

Q1Q2 1.93(<.001) *** 0.50 2.17(<.001) *** 0.52

Q2Q3 1.95(<.001) *** 0.56 2.09(<.001) *** 0.56

Q3Q4 1.11(<.001) *** 0.56 1.18(<.001) *** 0.55

Q1Q2Q3 2.11(<.001) *** 0.59 2.36(<.001) *** 0.63

Q2Q3Q4 2.04(<.001) *** 0.62 2.16(<.001)*** 0.62

Q1Q2Q3Q4 1.89(<0.001)*** 0.63 1.64(<0.001)*** 0.70

46

Generalizability

End-of-YearGrade

Samples β(p-value) R2

Q1 .345(.17) 0.621Q2 0.303(.43) 0.612Q3 0.450(.31) 0.615Q4 0.283(.18) 0.612

Q1Q2 0.259(.16) 0.618Q2Q3 -0.618(.54) 0.612Q3Q4 0.206(.46) 0.616

Q1Q2Q3 0.200(.26) 0.615Q1Q2Q3Q4 0.177(.30) 0.619

Repeated predictive analysis with geometry tutor

47

Predictive ValidityEnd-of-YearGrade UnitsCompleted

Samples β(p-value) R2 β(p-value) R2

Q1Q2Q3Q4 1.89(<0.001)*** 0.63 1.64(<0.001)*** 0.70Q1Q2Q3 2.11(<.001) *** 0.59 2.36(<.001) *** 0.63Q2Q3Q4 2.04(<.001) *** 0.62 2.16(<.001)*** 0.62Q1Q2 1.93(<.001) *** 0.50 2.17(<.001) *** 0.52Q2Q3 1.95(<.001) *** 0.56 2.09(<.001) *** 0.56Q3Q4 1.11(<.001) *** 0.56 1.18(<.001) *** 0.55Q1 1.85(<.001)*** 0.40 2.23(<.001) *** 0.43Q2 1.69(<.001)*** 0.45 1.72(<.001) *** 0.41Q3 1.08(<.001) *** 0.45 1.16(<.001) *** 0.44Q4 2.52(<.001) *** 0.51 2.09(<.001) *** 0.39

48

Documents

Detecting Diligence with Online Behaviors on Intelligent ...learningatscale.acm.org/las2017/wp-content/uploads/... · Online Behaviors on Intelligent Tutoring Systems Steven Dang