A progress report--Computer coaches for introductory physics
problem solving
Qing Xu, Ken Heller, Leon Hsu, Andrew MasonUniversity of Minnesota
Supported by NSF DUE #0230830 and DUE #0715615 and by the University of Minnesota
10/30/2010 MAAPT WAPT
Computer coaches for Introductory Physics
• Enhance Problem-Solving skills
• Possible Advantages:
- Provides individualized guidance and feedback outside of class; on-demand help
- Effectiveness through good design and pedagogy
- Customizable by instructors
Research Background
• Theory:
- Cognitive Apprenticeship1
• Design:
- Reciprocal teaching2
- Context-rich problems3
3 types of coaches
• 1st: Computer coaches the student– computer decides, student implements, computer assesses
• 2nd: Student coaches the computer– Student decides, computer implements, student assesses
• 3rd: Student works more independently– Computer provides help as necessary; Scaffolding
gradually withdraw
PILOT STUDY –Fall 2010• ~66 students, 1 lecture session of intro calculus-based physics
• Assign into 3 statistically matched groups (~22 for each group)
• Variables for matching: background information, e.g. HS physics & math level, FCI/CLASS/math pretests
• Subset of tutors available – energy, momentum (4 weeks)
• 1 Treatment group , 1 Control group and 1 placebo group• Treatment- computer coaching (on Web, outside of class), 4 problems per week • Placebo group – Work on Computer tutor problems on paper• Control group -- normal class setting
• Data collection• Written solutions on quizzes & final exam • 2×4+5=13 for each student
Today’s topics • Can written solutions well represent their Problem-
Solving skills? -- Yes.
• Interviews – (methodological triangulation) -- Triangulation: the use of two or more methods of data collection in the study of some aspect of human behavior. 4
-- An important source of evidence for validity. 5
• How to score the written solutions?-- Problem-solving Rubric. 5
UD: Useful DescriptionPA: Physics ApproachSA: Specific ApplicationMP: Mathematical ProcedureLP: Logical Progression
• Rubric Training– Inter-rater reliability
INTERVIEWS – Fall2010 5 volunteering students from introductory calculus-based mechanics course for scientists and engineers
Interviews were taken place around week 6 (Dynamics)
1st-Kinematics/2nd-Dynamics/3rd –Kinematics
In an interview, students are asked to solve physics problems while their actions and voice are recorded (30mins). After completing the problem, they are asked to explain their reasoning to an interviewer (30mins).
TriangulationTriangulation
student written workstudent written work self-reported thought processes
self-reported thought processes
correspond
correspond
5 rubric category processes
5 rubric category processes
self-reported problem-solving
processes
self-reported problem-solving
processes
characterize
characterize
Rubric scores Rubric scoresproblem-solving skills
from other measures of their performance
problem-solving skills from other measures of
their performance
inferences
inferences
Chart1: Triangulation 5
General information
1 student actually talked out loud while he/she was working on the problem
1 out of 5 students finished all 3 problems (1st-Kinematics/2nd-Dynamics/3rd –Kinematics)
1 student asked for help during problem solving process
3 out of 5 students gave up on Problem1 after the first 10 minutes; 1 student was
persistent until getting a final result (15mins); 1 student got stuck and moved on the the
2nd problem(15mins)
The students didn’t use the equation sheet too much, they either saw the equations on it
but remembered it anyway, or they just ignored the equation sheet.
ExpectationsExperts
Problem solving is a process of making a series of judicious decisions
Framework• Create useful description• Plan solution based on general
principles• Carry out plan • Evaluate solution (also
intermediate steps)
Novices
Plunge directly into mathematical calculations
Do not have a very useful or complete description (picture)
Focused on the superficial features of the problem
Transcripts/interview results
S6: The first thing I did is the angle of 30°, the initial velocity. I broke it down into components. Whenever there is an angle, I would do that first, so I don’t have to worry about it later on.
I: After I gave you the problem, What was the first thing that you did?
S6: Well, then I read the problem the second time. tried to absorb all the information. And, actually, I started to do my equations and backtracked and say that i don’t really know what the problem was saying. So I draw picture to kind of help me visualizing it. I kind of jumped into cracking some numbers, and then I realized, oh , wait, I don’t really get what the problem was asking, I actually go back to draw a little picture to help me visualize.
I: Okay, after that, what did you do?
interview results
S6: so I went back to work on the equations, using the equation sheet.
I: After you drew the picture, what did you do next?
Not a bad picture
Not complete
Not complete
Rubric Training & Scoring—Summer 2010
• 8 sets of previously scored student solutions• 1 undergraduate and 2 graduate students of physics• 1 experienced assessor and 2 novice assessors.
• Goal: For novice assessors to achieve an increased understanding of problem solving with measurable agreement.
Training
Qing and Talia
0
0.2
0.4
0.6
0.8
1
1.2
P1 P2 P3 P4 P5 P6 P7 P8
Solution Set
Agr
eem
ent w
ithin
1Agreement between Raters overtime
Figure 2: Agreement within 1 Rate 1& 36
Figure 1: Agreement within 1 Rate 1& 26
Qing and Karl (Before)
0
0.2
0.4
0.6
0.8
1
1.2
P1 P2 P3 P4 P5 P6 P7 P8
Solution Set
Agr
eem
ent w
ithin
1
Rater 1& 2 Rater 1& 3
Rubric Training & Scoring—Summer 2010
• 20 Students and 13 written solutions/student (4 quizzes & finals)• 1 undergraduate and 1 graduate students of physics• 2 assessors who completed training.
• Goal: Establish a baseline for future stages of research trends, dependencies( topics, gender, etc)
Scoring
Gender Gaps?
0
1
2
3
4
5
UD PA SA MP LP
Ave
rag
e R
ub
ric
Sco
re
Category
Quiz 1
Female
Male
Figure 5: Average score in each category for male and female students. Data is for the first quiz of the semester , however no statistically significant gender gap is apparent in any of the tests throughout the semester nor in student improvements from the first quiz to the final. 7
Improvement Overtime?
Figure 6: Changes in average rubric scores for each category from the beginning to end of the semester. Quiz 1 Problem 1 and Final Exam Problem 2 test the same physics concepts. Significant (p<0.05) gains are seen only in Physics Approach between Quiz 1 Problem 1 and Final Exam Problem 2. 7
References
1. Collins, Brown, & Newman, 1989. Cognitive apprenticeship.
2. Palincsar, A.S., & Brown, A.L. (1984). Reciprocal teaching of comprehension- fostering
and monitoring activities. Cognition and Instruction, 1, 117-175.
3. Heller, P., & Hollabaugh, M. (1992). Teaching problem solving through cooperative
grouping. Part 2: Designing problems and structuring groups. American Journal of Physics,
60(7), 637-644.
4. Louis Cohen, Lawrence Manion, Keith Morrison, Keith R. B. Morrison, (2007). Research
methods in education.
5. Docktor, 2009; Docktor & Heller, 2009
6. Talia Clark, Rubric Training Assessment Presentation, Summer 2010
7. Talia Clark, PER Final Presentation, Summer 2010