Effective Help-seeking Behavior

Effective Help-seeking Behavior Among Students Using an Intelligent Tutoring System for Math: a Cross-Cultural ComparisonJose Carlo A. SorianoBreak downEffective Help-seeking Behavior Among Students

Using an Intelligent Tutoring System for Math:

A Cross-Cultural ComparisonIntelligent Tutoring SystemsSeeks to simulate the effectiveness of a good personal human tutorIndividual tutoring is more effective than classroom instruction by 2 standard deviations (Bloom, 1984)Knows what specific skills the student is having trouble withAble to offer appropriate help at an appropriate time

Objective: help the student learnITS are better by 1 standard deviation (Koedinger et al, 1998; Corbett et al., 2001)Intelligent Tutoring Systems

Help-Seeking in ITSThe kind of help, and how the help is offered, affects learning (Aleven et al, 2003)Students who use High-level help most frequently have the least learning(Matthews and Mitrovic, 2008)

Help-seeking is a Meta-cognitive skillMeta-cognition is cognition about cognitionOnes knowledge of the processes in playOnes active control of it during learningHelp-seeking BehaviorBoth the EU and UNESCO declared:developing metacognitive skills, or teaching students how to learn should be among the highest educational priorities (Louizidu and Kotselini, 2007)

Help-seeking behavior is an achievement-related behavior (Karabenick and Knapp, 1991)

Higher-achieving students were more likely to ask for help when encountering personal difficulties (Taplin et al, 2001)Effective Help-SeekingStudent is more likely to learn when:Student seeks for help when encountering personal difficultiesStudent knows what kind of help is needed such that student can work effectively on his/her ownStudent knows how to ask for helpStudent is not dependent on helpStudent spends time understanding help givenHelp-seeking in ITSHowever, students generally do not know when they need help (Aleven and Koedinger, 2000)Students game the system (Baker et al)

Meta-cognitive tutors have been developed by Aleven et alScooter the tutor developed by Baker et alTo teach students how to learnThe ProblemIs effective help-seeking the same across cultures?Very few cross-cultural comparisonsComparing ITS use between USA and Latin American students(Ogan et al, in press)Comparing Disengaged behavior between USA and Filipino students (Rodrigo et al, 2010)Implications on Meta-cognitive tutorsMethodFind out if effective help-seeking behavior is the transferrable across cultures, or are significantly differentMight encourage more cross-cultural comparisonsImplications on future efforts on meta-cognitive tutoringScatterplot tutor

Data

Costa Rica

Mexico

USA

PhilippinesFeature Engineering1. Helpavoidance2. Nothelpavoidance3. Helpnonuse4. Unneededhelp5. BugmsgLongpause6. BugmsgShortpause7. HintmsgLongpause8. HintmsgShortpause

Feature Engineering9. HintmsgLongpauseCorrect10. HintmsgShortpauseCorrect11. NothelpavoidanceShortpause12. NothelpavoidanceLongpause13. UnneededhelpShortpause14. UnneededhelpLongpause15. ShortpauseHintmsg16. LongpauseHintmsg17. FirstattemptHintmsgFeature OptimizationMost features require a threshold, either p-know or a time threshold

Brute-force grid search:For p-know thresholds, grid-size is 0.05For pause thresholds, grid-size is 0.5 secondsSingle-parameter linear regression for each threshold for each feature in grid

Feature SelectionCross-validated r was used as the goodness criterionCorrelation between the predicted learning values and the actual learning value

The threshold with the best cross-validated r becomes the threshold for each feature

As an additional control against over-fitting, features whose best threshold had negative cross-validated r is dropped from model creationModel Creation and EvaluationModels were created using Forward Selection

Models were evaluated by applying each countrys model to each countrys data set

A model were created after combining the four data sets

Brute-Force Grid SearchFeaturecut-offrFeaturecut-offRCRPHHelpavoidance0.150.081Nothelpavoidance0.40.087Helpnonuse0.150.012Helpnonuse0.950.043Unneededhelp10.006NothelpavoidanceShortpause10.108BugmsgLongpause25.50.06NothelpavoidanceLongpause00.075HintmsgLongpause47.50.054UnneededhelpShortpause0.50.061HintmsgShortpause0.50.017USHintmsgLongpauseCorrect41.50.294Helpavoidance0.250.122NothelpavoidanceLongpause45.50.284Helpnonuse10.014UnneededhelpShortpause130.019Unneededhelp10.117UnneededhelpLongpause00.008BugmsgLongpause570.131MXBugmsgShortpause0.50.026Helpnonuse10.201HintmsgLongpause0.50.003BugmsgShortpause2.50.044HintmsgShortpause6.50.02HintmsgShortpauseCorrect0.50.025HintmsgLongpauseCorrect10.039NothelpavoidanceLongpause58.50.018HintmsgShortpauseCorrect40.073ALLNothelpavoidanceShortpause0.50.265Helpavoidance0.050.023NothelpavoidanceLongpause00.096Nothelpavoidance0.40.024UnneededhelpShortpause120.203Helpnonuse00.094UnneededhelpLongpause00.123BugmsgShortpause2.50.071ShortpauseHintmsg32.50.026HintmsgLongpauseCorrect10.022LongpauseHintmsg37.50.04HintmsgShortpauseCorrect30.062NothelpavoidanceShortpause200.027NothelpavoidanceLongpause10.052UnneededhelpShortpause350.04UnneededhelpLongpause10.062AnalysisFirstattemptHintmsg only feature that was not able to pass in any countryDifferent nature of skillsSome can be hard at the very start, some can be easily understood from the startMost skills may require a large number of actions

NothelpavoidanceLongpause had positive cross-validated correlation in all five data setsIn contrast to theory, this feature has negative directionality for most countriesThis might be because students who exhibit the behavior simply do not understand the skillForward SelectionCountryLearning =Cross-validated rCR0.132 * Helpavoidance(0.15)+ 7.385 * HintmsgLongpause(47.5)- 9.096 * HintmsgLongpauseCorrect(41.5)- 21.847 * NothelpavoidanceLongpause(0.25, 45.5)+ 53.0100.462MX - 0.147 * Helpnonuse(1) - 0.754 * BugmsgShortpause(2.5)+ 1.187 * HintmsgShortpauseCorrect(0.5)+ 40.6520.229PH 0.021 * Helpnonuse (0.95) - 0.763 * NothelpavoidanceShortpause(0.4, 1)+ 32.4230.126US- 1.021 * Nothelpavoidance(0.25) - 2.870 * BugmsgLongpause(57) - 6.680 * NothelpavoidanceShortpause(0.25, 19.5) + 5.605 *LongpauseHintmsg(37.5) + 12.0860.350ALL0.036 * Helpavoidance(0.05) + 0.082 * Helpnonuse(0) - 0.491 * BugmsgShortpause(2.5) - 46.8610. 153AnalysisHelpavoidance negative directionality for CR and PHreinforcces Aleven et als findings that avoiding help is negatively correlated to learning (Aleven et al., 2006)

ButmsgShortpause and NothelpavoidanceShortpause also has negative directionality for MX and US, and PH and US.Possibly effect of not spending enough time to understand bug or hint message provided

Cross-Cultural EvaluationCountryCRMXPHUSALLCR0.534-0.2850.0510.1510.08MX0.040.392-0.086-0.0090.088PH0.004-0.1740.2030.1460.038US-0.085-0.1640.2280.4760.057All-0.0320.1650.0470.1420.215Rows are models, columns are data sets applied toAnalysisEighteen out of 25 model applications produced positive correlation between models predicted learning and the actual learningNegative correlation means model did worse than chance at predicting learningLOOCV r values are very high compared to r when applied to other countriesReinforces hypothesis that help-seeking might not transfer across countriesAnalysisMX and US performance on each others data sets are low (-0.009 and -0.164)This means that our model of effective help-seeking is not effective when applied to the other countryReinforces findings in (Ogan et al, in press) which compares differences in behavior of USA and Mexico studentsCollaborative nature of students from Mexico may be the reason why help-seeking is differentThe help they ask from the tutor will only be help that they do not get from other studentsAnalysisCR and MX did not perform well on each others data sets (-0.285, 0.04)Though collaborative tendencies might be common between Costa Rica and Mexico, help-seeking behavior with the ITS may differUS and PH performed very well on each others data set (0.146 and 0.228)Reinforces findings in (San Pedro, 2011) wherein a carelessness detector is generalizable between the two countriesIn contrast to (Rodrigo, 2010) which shows disengaged behavior is different between the two countriesBut Help-seeking and Disengaged behavior are two different sets of behaviorsConclusionResults did not expose that effective help-seeking as a whole is very culture-specific (18 out of 25 applications returned positive r)

However, it is not apparent that effective help-seeking is transferrable across countriesBig difference between LOOCV r and cross-cultural evaluationsConclusion and RecommendationsThere are pairs of countries wherein effective help-seeking fail to generalize to each others data set.Meaning effective help-seeking from one country may not necessarily be effective in another

Single effective help-seeking models used by meta-cognitive tutors may be effective in one culture but not in othersFuture meta-cognitive tutors might have to use a more generalizable modelMay have to switch models, depending on the culture where the ITS is used