Context Personalization, Preferences, and Performance in an Intelligent Tutoring System for Middle School Mathematics

8/12/2019 Context Personalization, Preferences, and Performance in an Intelligent Tutoring System for Middle School Mathe…

http://slidepdf.com/reader/full/context-personalization-preferences-and-performance-in-an-intelligent-tutoring 1/5

Context Personalization, Preferences, and Performance inan Intelligent Tutoring System for Middle School

MathematicsStephen E. Fancsali

Steven RitterCarnegie Learning, Inc.

437 Grant Street, Suite 918

Pittsburgh, PA 15219, USA

888.851.7094 {x219, x122}

[email protected]@carnegielearning.com

ABSTRACT

Learners often think math is unrelated to their own interests.

Instructional software has the potential to provide personalizedinstruction that responds to individuals’ interests. CarnegieLearning’s MATHia

TM software for middle school mathematics

asks learners to specify domains of their interest (e.g., sports &fitness, arts & music), as well as names of friends/classmates, anduses this information to both choose and personalize word

problems for individual learners. Our analysis of MATHia’srelatively coarse-grained personalization contrasts with more fine-

grained analysis in previous research on word problems in theCognitive Tutor (e.g., finding effects on performance in parts of

problems that depend on more difficult skills), and we exploreassociations of aggregate preference “honoring” with learner

performance. To do so, we define a notion of “strong” learnerinterest area preferences and find that honoring such preferenceshas a small negative association with performance. However,learners that both merely express preferences (either interest area

preferences or setting names of friends/classmates), and those thatexpress strong preferences, tend to perform in ways that are

associated with better learning compared to learners that do notexpress such preferences. We consider several explanations ofthese findings and suggest important topics for future research.

Categories and Subject Descriptors

K.3.1 [Computers and Education]: Computer Uses in Education – Computer-assisted instruction (CAI).

General Terms

Measurement, Performance, Design, Human Factors.

Keywords

Non-cognitive factors, intelligent tutoring systems, preferences, personalization, mathematics education.

1. INTRODUCTIONIntelligent tutoring systems (ITSs) like Carnegie Learning’s

Cognitive Tutor ®

(CT) [7] adapt to learners’ evolving knowledge by tracking their performance as opportunities to practice particular knowledge components (KCs), or skills, are

encountered. CTs probabilistically assess learner mastery of KCsand dynamically present problems based on KCs a learner has yetto master. The educational effectiveness of adapting to cognitivefactors in this manner is well established [6][9], and many recentefforts have focused on enhancing both the educational and

motivational effects of the software by personalizing instruction based on non-cognitive factors [2][4][12].

In this paper, we focus on context personalization, wherein“features of an instructional program are matched to individuallearner’s personal interests and experiences” [11]. CarnegieLearning’s MATHia

TM software is a CT-based ITS for middle

school mathematics which asks learners to specify domains oftheir interest (e.g., sports & fitness, arts & music) and uses thisinformation to pick word problems for individual learners.

MATHia also provides the ability for learners to specify names offriends or classmates, which become the names of charactersappearing within problems. Both personalization on content andnames have been shown, in prior experiments, to improvelearning outcomes (e.g., improved problem solving, motivation,and engagement) [1][3][5].

Walkington [12] found, in an experimental version of CT, that

content personalization improved learning outcomes, particularlyon difficult problem steps and on problems with high readinglevel. This work was done on a small number of instructionalunits, which were completed by learners in a few weeks. Ourfocus here is to look at the long-term sustained impact of

personalization across a whole school year’s worth of curriculum.We are particularly concerned with how learners react to the

extent to which the system “honors” their preferences for particular topics and with whether learners expressing strong preferences for a topic (or preferences at all) perform differentlythan those who have weak preferences or who did not specifyinterest area or name preferences.

2. LEARNER PREFERENCES IN MATHIABuilding on the CT approach to mathematics curricula likealgebra, MATHia (Figure 1) provides an environment directed atyounger learners in a series of three middle school mathematics

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that

copies bear this notice and the full citation on the first page. Copyrightsfor components of this work owned by others than ACM must behonored. Abstracting with credit is permitted. To copy otherwise, or

republish, to post on servers or to redistribute to lists, requires priorspecific permission and/or a fee. Request permissions [email protected].

LAK '14, March 24 - 28 2014, Indianapolis, IN, USACopyright 2014 ACM 978-1-4503-2664-3/14/03…$15.00.http://dx.doi.org/10.1145/2567574.2567615



courses. In particular, MATHia introduces several non-cognitive,context personalization features.

Figure 1. Screenshot of typical, non-personalized algebra

problem solving in MATHia, with a worksheet/table to

provide answers to parts of the problem scenario and a graph

to plot points (© Carnegie Learning, Inc.)

MATHia contextually personalizes word problems to learner

preferences in two ways. First, MATHia allows learners to rate

four interest area domains (on a scale of one to five “stars”) forwhich mathematics word problems have been tailored: sports &fitness, money & business, arts & music, and environment &nature1. Second, MATHia allows learners to set names of theirfavorite classmates or friends that can be integrated into thelanguage of problems. Learner preferences are optionally set viaa profile, illustrated in Figure 2. One reason learners may not set

preferences is that they do not discover this profile (whether viaexploration in the software, their peers, or instructors).

Figure 2. MATHia preferences profile

(© Carnegie Learning, Inc.)

Problem assignment in MATHia proceeds in much the same wayas in the CT; problems are presented that maximize the learner’sexposure to KCs of which the CT has not judged learner mastery.However, assuming a learner has provided preference ratings forone or more interest areas, MATHia probabilistically chooses

problems that also may honor those preference settings. If two problems emphasize the same KCs, that set is the maximal set ofKCs the learner has yet to master in a particular section, and one

problem has been written to correspond to an interest area that thelearner has rated higher than the interest area of the other problem(or that problem is not preference-tailored), then MATHia is more

1 A fifth area, humor (i.e., comical problem text), is assumed to bean interest area of all learners. We leave analysis of problemstailored to this interest area for future work.

likely (but not guaranteed) to present the problem tailored to theinterest area with the higher rating (or any rating at all).

3. DATATo summarize overall preferences, we considered a snapshot of adatabase that stores learner preferences in MATHia. Thissnapshot contained records for 104,197 learners that logged intoMATHia at least once from its release in 2011 until the middle of2013. Of these learners, 59.7% (62,168) set interest area

preferences. Among those with interest area preferences, 85.2%(52,938) provided names of friends or classmates.

For detailed analysis of associations between preference settingsand learner performance, we consider data for 2,389 learners,from eight

2 randomly selected middle schools that have a

substantial number of MATHia users, with activity in at least oneMATHia section that includes preference-tailored problems

3.

Specifically, we focus on a set of 1,230 learners that “graduated”from (i.e., mastered all KCs associated with) at least five

preference-tailoring sections. On average, these learners werelogged into MATHia for 7.4 hours while working throughsections that include preference-tailored word problems. Of theselearners, 65.4% set interest area preferences (805 learners) and71.7% (875 learners) provided names of friends and classmates.

3.1 Summary of Learner Interest AreasFrom our overall sample of learners that set preferences inMATHia, we find that the five star rating is most common, andthat learners most frequently rate sports & fitness with five stars,followed next by arts & music. Figure 3 provides overall countsfor each interest area of the ratings they have received.

Figure 3. Frequencies of MATHia interest areas ratings

Two observations lead us to define a more informative notion oflearner preferences:

• 15.7% of learners with preference settings (9,786learners) rate all interest areas with five stars.

• 25.7% of learners with preference settings (16,003learners) rate all interest areas with the same rating.

Learners that rate all interest areas with the same number of starsmay have little interest in any interest areas; diverse interests in allareas; or, perhaps these learners are not meaningfully providinginformation about their interests. Since we are not in a position to

disentangle these possibilities, we provide a definition of “strong” preferences and posit that, for learners whose preferences meet

2 Processing fine-grained log data for detailed analyses is time-consuming, and we have yet to process data for all MATHiausers. Nevertheless, we have no reason to believe that thissubset is not representative of our broader user population.

3 Roughly 20% of the 300 sections across all three MATHiacourse curricula include preference-tailored word problems.



this definition, we have informative data to analyze relationships between “honoring” preferences and learning.

3.2 “Strong” PreferencesFor MATHia interest areas, we define learners to have “strong”

preferences if they set at least one interest area with a five starrating and at least one interest area with a one star rating or norating. We posit that such learners provide more informative

preferences and that these are individuals for whom the notion of

honoring preferences is clear.By this definition, 61.2% (38,060) of learners in our large samplewith set interest areas have strong preferences. In our smallersample for detailed analysis, 64.3% of learners (805) with setinterest areas have strong preferences.

3.3 Learning OutcomesRecent analysis of CT usage data for a sample of 3,224 middleschool learners in a Virginia school district [8] provides a set ofCT usage or “process” variables that are positively associatedwith scores on the Virginia Department of Education’s Standards

of Learning (SOL) mathematics exam [10]. Since we lackexternal measures like standardized test scores or final examscores for our sample of MATHia learners, we use these variablesas learning outcomes.

We construct the following variables over MATHia sections thatinclude preference-tailored word problems from log data for oursample of 1,230 learners that graduated from at least five suchsections:

• assistance_per_problem: average over sections of the

per-section average sum of hints and errors per problem(log-transformed and normalized per section); this is anindicator of struggle during problem solving.

• sections_encountered : log-transformed number ofsections that a learner attempts that contain preference-tailored problems in MATHia4

• sections_mastered_per_hour : log-transformed average

number of sections from which a learner graduates per

hour; this is an indicator of efficiency in workingthrough material.

• total_login_time: total amount of time the learner waslogged into MATHia in sections that include

preference-tailored problems

Given the effectiveness of CT [6] and past results [8], studentsencountering more material and proceeding efficiently through it

provide reasonable outcomes, absent external measures. We nowexplore associations between the extent to which we “honor”

preferences and these learning outcomes. Then we consider howlearners’ mere expression of preferences is associated with theseoutcomes.

4. ANALYSIS

4.1 “Honoring” Preferences & PerformanceSince MATHia probabilistically presents problems that matchlearner interest areas when a set of problems all containappropriate KCs that a learner has yet to master, we can expect,

by chance, different learners to experience different rates of

4 We diverge from [8] and do not include a variable representingthe number of skills encountered in sections of MATHia

because this variable is almost perfectly correlated (r = .99) with sections_encountered .

experiencing problems that correspond to their preferences. Wecall cases where MATHia presented a problem corresponding tothe learner’s preference “honoring” that preference. For learnersthat set strong preferences, we calculate the proportion of

problems they were presented that match their five star interest

areas, in sections that include preference-tailored problems. Wefind that there is a relatively restricted range for this proportion,illustrated in the histogram for 518 learners in Figure 4. Even insections that include problems tailored to interest areas, most

learners are presented problems that match their interest area(s)only 10% to 30% of the time. While the proportion of preference-honoring problems assigned is randomly determined per learner,making this a natural experiment, the restricted range in thisdistribution must be kept in mind in interpreting results.

Figure 4. Histogram of strong preference learners (n = 518)

counts & the proportion of preference (i.e., interest area)

honoring problems presented

Despite this restricted range, we consider correlations of honoringinterest areas for strong preference learners with the four learningoutcomes we have chosen. To illustrate, Figure 5 is a scatterplotof assistance_per_problem and the proportion of preferencehonoring problems.

Figure 5. Scatterplot of assistance_per_problem & proportion

of preference honoring problems for strong preference

learners (n = 518)

Visual inspection of this scatterplot does not lead to an impressionof a strong correlation or association between assistance (i.e., theextent to which a learner struggles) and honoring preferences.Table 1 provides quantitative pairwise correlations between

preference honoring and our four learning outcomes and showsPearson’s r = .14 in the scatterplot of Figure 5. While three of thefour variables have statistically significant correlations with

preference honoring, none of the correlations are especially large.

Further, controlling for learner efficiency via sections_mastered_per_hour , we find that the partial correlation



of assistance_per_problem and preference honoring isinsignificant (r = .043, p = .33) as is the correlation between

preference honoring and total_login_time (r = .063, p = .154).This is partially due to expected dynamics of learner interactionwith the tutor and the relatively high correlation of sections_mastered_per_hour with assistance_per_problem (r =-.76, p < .001) and with total_login_time (r = -.69, p < .001) inthis sub-sample

5.

Table 1. Pairwise Pearson correlation coefficients for each

MATHia process variables and the proportion of preference

honoring problems presented to learners

!"#$%& ()*+,-- .&)%&/0, 1,&)-*23- ) ( .&04,

!""#"$!%&'()'*()*+,-'. !"# !%%&

"'&$#+%"('%&+0%$'*'1 !%' !%(&

"'&$#+%"(.!"$'*'1()'*(2+0* )!"# !%%"

$+$!-(-+3#%($#.' !"# !%%"

These analyses are not intended to exhaust the investigation of thedynamics among our chosen learning outcomes, but merely tosuggest that the aggregate effect (if any) of preference honoringon such coarse-grained learning outcomes is not large. We

provide one possible explanation of a weak negative association

between preference honoring and learning in the discussion.However, given a lack of a strong association, we focus on the

broader issue of whether merely expressing preferences isassociated with better learning outcomes.

4.2 Strong vs. Weak Interest Area

Preferences & PerformanceFirst, we compare differences in learning outcomes for learnersthat set strong interest area preferences (n = 518) compared tothose that set weak preferences (n = 287).

Table 2. Table of Welch two sample, two-sided t-test results

comparing the mean of MATHia process variables for

learners with strong vs. weak interest area preferences

!"#$%& ()*+,-- .&)%&/0, -5)*26

7,&28-9:

;,&<

7,&28-9:

( 9

!""#"$!%&'()'*()*+,-'. )!"(*!(+,

)!%-*!+%,

!%( )!"#

"'&$#+%"('%&+0%$'*'1 &!+-*!#+,

&!+&*!##,

!"" !""

"'&$#+%"(.!"$'*'1()'*(2+0* !-+*!+&,

!+.*!+(,

!%" !&

$+$!-(-+3#%($#.' -!"*#!&,

-!+*#!',

!"+ )!"

Results for two-sided Welch two sample t-tests for difference inmeans as well as values of Cohen’s d are provided in Table 2. We

provide calculations of d only for perspective of how

5 Both correlations are somewhat smaller in the larger sample of1,230 learners that graduated from at least five preference-honoring sections: for assistance_per_problem, r = -.69 (p <.001) and for total_login_time, r = -.59 (p < .001).

6 Recall that assistance_per_problem is normalized (i.e., a z-score) across all learners that graduated from at least 5 sectionswith preference-tailored problems.

7 We provide mean total_login_time in hours.

“substantive” significant differences are, as we are working withobservational data and with non-trivial sample sizes.

We find one statistically significant difference at the ! = .05 level:learners with strong preferences tend to work through MATHiasections that include preference-tailored problems more efficiently(i.e., have greater sections_mastered_per_hour ). As greaterefficiency is positively associated with an external standardized

test in a previous study, we might predict that learners with strong preferences may achieve better outcomes as well.

Strong preference learners tend to require assistance less than

those with weak preferences, but the difference is not statisticallysignificant. Further, weak preference learners tend to encounterfewer sections with greater time logged into MATHia sections,

but neither difference is statistically significant.

4.3 Interest Area Preference Settings &

PerformanceWe now consider differences in MATHia process variables

between learners that set interest area preferences (both strong andweak, n = 805) versus those learners that did not set interest area

preferences (n = 425).

Table 3 provides a summary of these results. We find that the

only significant difference is that learners that set preferences seek(or require) less assistance. While on average they encounter

roughly the same number of sections, working through materialwith slightly greater efficiency while logged in for less time, noneof the differences are statistically significant. Sinceassistance_per_problem is negatively associated with externallearning outcomes [8], we would predict that those who merelyset their interest area preferences also achieve better outcomes.

Table 3. Table of t-test results comparing the mean of

MATHia process variables across learners that set interest

area preferences vs. those that did not set preferences

!"#$%& ()*+,-- .&)%&/0, -,5

7,&28-9:

2*5 -,5

7,&28-9:

( 9

!""#"$!%&'()'*()*+,-'. )!"&*!(/,

)!%(*!+%,

!%. )!"&

"'&$#+%"('%&+0%$'*'1 &!+(*!#(,

&!+-*!#(,

!./ )!%#

"'&$#+%"(.!"$'*'1()'*(2+0* !-"*!+.,

!++*!++,

!"' !%/

$+$!-(-+3#%($#.' -!.

*#!(,

-!+

*#!+,

!&# )!%-

4.4 Setting Names & PerformanceWe now consider whether learners’ providing names ofclassmates is associated with better learning. Table 4 presents thesame statistics we have considered but for learners that set names

of classmates or friends (n = 875; of these learners, 721 set

interest area preferences) versus those learners that do not do so (n= 355; 84 of whom set interest area preferences).

We find that the direction of differences align with trends weobserve for learners with strong preferences and those that setinterest areas, but we find that these differences are all significantwhen compared to learners that did not set names. Learners thatset these names encounter more material while completingsections more efficiently (and with less assistance) in less time.That is, overall, they perform better in MATHia on metrics thathave been found to be associated with external learning outcomes.



Table 4. Comparison of learners that set names of classmates

or friends vs. learners that do not do so

!"#$%& ()*+,-- .&)%&/0, -,5

7,&28-9:

2*5 -,5

7,&28-9:

( 9

!""#"$!%&'()'*()*+,-'. )!"#*!(-,

!%&*!+",

0!%%" )!&-

"'&$#+%"('%&+0%$'*'1 &!+/

*!#(,

&!+"

*!##,

!%%' !"+

"'&$#+%"(.!"$'*'1()'*(2+0* !-(*"!-,

!(+*"!(,

0!%%" !"&

$+$!-(-+3#%($#.' -!"

*#!&,

/!&

*(!&,

0!%%" )!&.

5. DISCUSSION & FUTURE WORKOur results raise important questions about effects of

personalization, metrics, and ITS design. These results suggestonly weak, possibly negative effects for honoring domain

preferences over a school year. On the other hand, we find some

advantages for learners who set such preferences and, amongthose, of learners who have strong preferences. With respect tospecifying names of characters, we find relatively robust effects.

In interpreting these results, it is important to consider thelearner’s perspective within MATHia. Only a subset of MATHia

problems are word problems that are amenable to the type of personalization we explore here. In fact, only 20% of sectionscontain word problems that could respond to these preferences.

Since preference honoring was probabilistic, the learner’sexperience would be that a small percentage of problems honor

preferences (even smaller than proportions shown in Figure 4).Combined with the restricted range of preference honoring, this isa fairly weak manipulation. In fact, learners may be frustrated thatspecifying their preferences has such a small effect on

performance, perhaps leading to the small negative correlation between the extent to which MATHia honors preferences and sections_mastered_per_hour. While honoring preferences more

frequently might counter this, it also might be possible to make preference honoring more visible in the MATHia interface. Forexample, future software might notify learners about conditionsthat have led to the presentation of a problem (e.g., “Because youlike sports, we chose the following…”).

The findings on setting domain preferences and names of friendsmay indicate that some learners are more appreciative of the

opportunity to set preferences, leading to better positive affecttowards MATHia and better outcomes. Alternatively, setting such

preferences may be driven by a factor such as conscientiousness.Learners who are predisposed to care about their work in mathclass may be more likely to set such preferences and more likelyto achieve better outcomes. Differences in the tendency to setnames vs. domain preferences may allow us to better explore this

possibility. Future work might also include a measure of priormath ability (lacking in our large but retrospective sample) and/orconfidence or self-efficacy to better understand this association.

Prior research [1][3][5][11][12] has shown that, at the problemlevel, we see advantages from personalizing problems. Theseexperiments typically personalize all problems presented over ashort period of time. To extend these results to everydayclassrooms, we need to understand whether this relativelyintensive personalization will scale to the context where learnersget personalized problems spread out over a full school year. Our

initial results may suggest that personalization at the problem

level needs to be more apparent to learners (either by making itmore frequent or by pointing out the personalization) but also thatthe use of personalization features appears to be associated withimproved outcomes. Future work will build on these possibilities.

6. ACKNOWLEDGMENTSWe thank Susan Berman, Bob Hausmann, Chas Murray, Tristan

Nixon, and Michael Yudelson for comments on this work.

7. REFERENCES[1] Anand, P.G., and Ross, S.M. 1987. Using computer-assistedinstruction to personalize arithmetic materials for elementaryschool children. J Educ Psychol 79, 1, 72–78.

[2] Bernacki, M.L., Nokes-Malach, T.J., and Aleven, V. 2013.Fine-Grained Assessment of Motivation Over Long Periodsof Learning with an Intelligent Tutoring System:Methodology, Advantages, and Preliminary Results. In International Handbook of Metacognition and LearningTechnologies. Springer, Berlin, 629-644.

[3] Cordova, D. I., and Lepper, M. R. 1996. Intrinsic Motivationand the Process of Learning: Beneficial Effects ofContextualization, Personalization, and Choice. J Educ

Psychol 88, 4, 715-730.

[4] Fancsali, S.E., Ritter, S., Stamper, J., and Nixon, N. 2013.Toward “Hyper-Personalized” Cognitive Tutors. In AIED

2013 Workshop Proc. Vol. 7: GIFT (Memphis, TN, July 13,2013), 71-79.

[5] Ku, H., and Sullivan, H. J. 2002. Student performance and

attitudes using personalized mathematics instruction. Educational Technology Research and Development 50, 1,21-34.

[6] Pane, J.F., Griffin, B.A., McCaffrey, D.F., and Karam, R.2013. Effectiveness of Cognitive Tutor Algebra I at Scale.RAND Education Working Paper: WR-984-DEIES.

[7] Ritter, S., Anderson, J.R., Koedinger, K.R., and Corbett, A.T.2007. Cognitive Tutor: Applied Research in Mathematics

Education. Psychon B Rev 14, 249-255.[8] Ritter, S., Joshi, A., Fancsali, S.E., and Nixon, T. 2013.

Predicting Standardized Test Scores from Cognitive TutorInteractions. In Proc. of the 6

th International Conf. on

Educational Data Mining (Memphis, TN, July 6-9, 2013).169-176.

[9] Ritter, S., Kulikowich, J., Lei, P., McGuire, C.L., andMorgan, P. 2007. What evidence matters? A randomizedfield trial of Cognitive Tutor Algebra I. In Supporting

Learning Flow through Integrative Technologies: Volume

162 Frontiers in Artificial Intelligence and Applications. IOSPress, Amsterdam, 13-20.

[10] Virginia Department of Education. 2013. Standards ofLearning (SOL) & Testing. Retrieved October 9, 2013.

http://www.doe.virginia.gov/testing/

[11] Walkington, C.A. and Maull, K. 2011. Exploring theAssistance Dilemma: The Case of Context Personalization.

In Proc. of the 33rd

Annual Meeting of the Cognitive ScienceSociety (Boston, MA, July 20-23, 2011). 90-95.

[12] Walkington, C.A. and Sherman, M. 2012. Using AdaptiveLearning Technologies to Personalize Instruction: TheImpact of Interest-Based Scenarios on Performance inAlgebra. In Proc. of the 10

th International Conf. of the

Learning Sciences (Sydney, July 2-6, 2012). 80-87.

Documents

Context Personalization, Preferences, and Performance in an Intelligent Tutoring System for Middle School Mathematics