Dr. Janice GobertPrincipal Investigator

jgobert@wpi.edu

Associate Professor,Social Sciences Dept. &

Computer Science Dept.

Co-Director, Learning Sciences & Technology Program

Science Assistments

Funded by NSF-DRL# 0733286, NSF-DGE #0742503, NSF-DRL #1008649) and by the U.S. Dept. of Education (R305A090170).

Science Assistments Team

InvestigatorsDr. Janice Gobert (PI), Social Sciences & Policy Studies, WPIDr. Neil Heffernan, Computer Science, WPIDr. Ryan Baker, Social Sciences & Policy Studies, WPIDr. Joe Beck, Computer Science, WPIDr. Carolina Ruiz, Computer Science, WPIDr. Ken Koedinger, HCII, Carnegie-Mellon University

Graduate Students & StaffArnon Hershkovitz, Ph.D, Post-Doctoral ResearcherErmal Toto, Ph.D. Student, LS&T/Software EngineerOrlando Montalvo, Ph.D. Student, LS&T/Software EngineerMichael Sao Pedro, Ph.D. Student, Computer ScienceJuelaila Raziuddin, Ph.D. Student, LS&TAdam Nakama, M.Sc. Student, LS&TMatt Bachmann, M.Sc. Student, Computer ScienceMike Wixon, M.Sc. Student, LS&TCameron Betts, M.Sc. Student, LS&T

Project Overview• Science Assistments is an

environment for conducting performance assessment of middle school students’ inquiry in Physical, Life, & Earth Science.

• Our activities are based on guided inquiry & experimentation with microworlds.

• We are auto-tutoring of students’ inquiry based on data mining and knowledge engineering.

Hello! You are going to be a scientist today and conduct experiments in a virtual laboratory!

Students learn and are assessed while they do inquiry with microworlds

• With microworlds, students:

- develop a hypothesis,

- design & conduct an experiment

- analyze data & warrant their claims, and

- communicate findings (NSES, 1996).

• And because we log all students’ actions, we can respond in real time using our pedagogical agent, Rex

Educational Data Mining to auto-analyze log files

• Extending prior work on using logs to characterize inquiry moves in microworlds (Buckley, Gobert et al, 2010).

• Extending work of Baker et al (2008) by applying text replays to label students’ inquiry moves.

Text Replay Tagging Software

Clip Tags

Student Clip

Replay

EDM, cont’d• Once logs are labeled, use EDM to

determine what fine-grained logged features correspond to specific inquiry skills.

• Build detectors over feature sets, i.e., aggregates of logged actions.

• Validate detectors (Sao Pedro et al, Montalvo et al, 2010).

“Goodness” of our detectors for coding students’ inquiry processes

• A’ = probability of correct labeling given 2 examples (+ and – examples)• Kappa = does the predictor do better than chance (chance level = 0; 1= perfect)

Sao Pedro et al., 2010; Montalvo et al., 2010

Testing Stated Hypothesis

A’ = 0.86 Kappa = 0.46

Using CVS during experimenting

A’ = 0.85 Kappa = 0.47

Data Table Use A’ = 0.94 Kappa = 0.46

Using Detectors to Predict Performance

Using our detectors as a basis for assessing authentic skill, we can generate models that let us:

(1)Predict skill proficiency before a student starts a new activity

(2)Research the relationship between authentic skill honed in our learning environment and other transfer measures of inquiry

Sao Pedro, Baker, Gobert, Montalvo, & Nakama (in prep.)

Results Models accurately predict next attempt on microworld for

each inquiry skill– Testing hypotheses (A’ = .79)– Designing controlled experiments (cvs) (A’ = .74)– Planning using the table tool (A’ = .71)

Authentic skill significantly correlates with performance on transfer measures of inquiry

– Multiple-choice “testing hypotheses” assessment (r = .41)– Multiple-choice “controlled experiments” assessment (r = .26)– Authentic “controlled experiments” assessment (r = .38)

Papers/results cited here Baker, R.S.J.d., de Carvalho, A. M. J. A. (2008) Labeling Student Behavior Faster

and More Precisely with Text Replays. Proceedings of the 1st International Conference on Educational Data Mining, pp. 38-47.

Buckley, B. C., Gobert, J., Horwitz, P., & O’Dwyer, L. (2010). Looking inside the black box:  Assessing model-based learning and inquiry in BioLogica. Int. Journal of Learning Technologies, 5(2), 166 - 190.

Montalvo, O., Baker, R.S.J.d., Sao Pedro, M.A., Nakama, A. & Gobert, J.D. (2010). Identifying Students’ Inquiry Planning Using Machine Learning. Proceedings of the 3rd International Conference on Educational Data Mining (pp. 141-150).

Sao Pedro, M.A., Baker, R.S.J.d, Montalvo, O., Nakama, A. & Gobert, J.D. (2010). Using Text Replay Tagging to Produce Detectors of Systematic Experimentation Behavior Pattern. Proceedings of the 3rd International Conference on Educational Data Mining (pp. 181-190).

Sao Pedro, M., Baker, R.S.J.d, Gobert, J., Montalvo, O., & Nakama, A. (in prep). Using Machine-Learned Detectors of Systematic Inquiry Behavior to Predict Gains in Inquiry Skills.

For more information & papers See our website, www.scienceassistments.org

Or contact Janice Gobert: jgobert@wpi.edu.