Analyzing the many skills involved in solving complex physics problemsWendy K. Adams and Carl E. Wieman Citation: American Journal of Physics 83, 459 (2015); doi: 10.1119/1.4913923 View online: http://dx.doi.org/10.1119/1.4913923 View Table of Contents: http://scitation.aip.org/content/aapt/journal/ajp/83/5?ver=pdfcov Published by the American Association of Physics Teachers Articles you may be interested in Light-Emitting Diodes: Solving Complex Problems Phys. Teach. 53, 291 (2015); 10.1119/1.4917437 Perimeter Institute for Theoretical Physics director and theoretical physicist Neil Turok's 2012 CBC MasseyLecture videos, “The Universe Within: From Quantum to Cosmos”, itunes.apple.com/ca/album/cbc-massey-lectures-2012-by/id577245484 Phys. Teach. 51, 190 (2013); 10.1119/1.4792030 Transfer of argumentation skills in conceptual physics problem solving AIP Conf. Proc. 1513, 322 (2013); 10.1063/1.4789717 Using classical probability functions to illuminate the relation between classical and quantum physics Am. J. Phys. 74, 404 (2006); 10.1119/1.2173280 Interactive video tutorials for enhancing problem‐solving, reasoning, and meta‐cognitive skills of introductoryphysics students AIP Conf. Proc. 720, 177 (2004); 10.1063/1.1807283

This article is copyrighted as indicated in the article. Reuse of AAPT content is subject to the terms at: http://scitation.aip.org/termsconditions. Downloaded to IP:

72.245.29.161 On: Tue, 02 Jun 2015 23:28:06

PHYSICS EDUCATION RESEARCH SECTION

The Physics Education Research Section (PERS) publishes articles describing important results from the

field of physics education research. Manuscripts should be submitted using the web-based system that can

be accessed via the American Journal of Physics home page, http://ajp.dickinson.edu, and will be forwarded

to the PERS editor for consideration.

Analyzing the many skills involved in solving complex physicsproblems

Wendy K. AdamsDepartment of Physics and Astronomy, University of Northern Colorado, Greeley, Colorado 80639

Carl E. WiemanDepartment of Physics and Graduate School of Education, Stanford University, Stanford, California 94305

(Received 6 January 2014; accepted 19 February 2015)

We have empirically identified over 40 distinct sub-skills that affect a person’s ability to solve

complex problems in many different contexts. The identification of so many sub-skills explains

why it has been so difficult to teach or assess problem solving as a single skill. The existence of

these sub-skills is supported by several studies comparing a wide range of individuals’ strengths

and weaknesses in these sub-skills, their “problem solving fingerprint,” while solving different

types of problems including a classical mechanics problem, quantum mechanics problems, and a

complex trip-planning problem with no physics. We see clear differences in the problem solving

fingerprint of physics and engineering majors compared to the elementary education majors that

we tested. The implications of these findings for guiding the teaching and assessing of problem

solving in physics instruction are discussed. VC 2015 American Association of Physics Teachers.

[http://dx.doi.org/10.1119/1.4913923]

I. INTRODUCTION

Problem solving is arguably the most important skill aphysicist can have, but in spite of extensive research, pro-gress has been quite limited in finding effective ways to mea-sure and teach this skill.1–3 Our hypothesis is that progress inproblem solving has been slowed by the difference betweenthe complexity of the subject and the simplicity of the mea-surement tools. Usually all that can be measured is whetherthe person can solve the problem or not. This provides noinsight as to what the learner needs to do to improve and noguidance to the teacher as to how to help them. It is like try-ing to teach a person to play golf when the only informationavailable is whether the ball they hit went into the hole ornot. “Just do it better” is an ineffective way to teach eithergolf or physics problem solving. “Go watch Tiger Woods”(or equivalently, an expert physicist) is only slightly morehelpful, because little of what such experts do is evident orreadily copied by non-experts.

Although numerous scholars have delineated and classi-fied different elements or sub-skills of problem solving, therewas little empirical work to justify that these were correctand complete characterizations and even less work on waysto measure an individual’s level of strength in each sub-skill.Assessing individual sub-skills, rather than rating a person’soverall “problem solving” ability, is necessary for teachersand researchers to understand their students’ current stateand to determine effective teaching strategies for improvingproblem solving.

In this paper, we provide a brief history of problem solv-ing research including work to teach problem solving skills,

characterization of the types of sub-skills that are required tosolve problems, and assessment of problem solving. Thisbackground is followed by a description of the researchmethodology we used to empirically identify the many dif-ferent sub-skills used when solving complex problems, thedevelopment of the rubric used to determine an individual’sstrengths and weaknesses in each sub-skill (their “problemsolving fingerprint”), and the different types of evidence col-lected to support the validity of these skills. This evidenceincludes expert review of the sub-skills and comparison of aperson’s problem solving fingerprint as determined whilesolving different types of problems including: in class prob-lem solving as observed by their physics instructor, a com-plex introductory-level mechanics problem; and quantummechanics problems. Finally, we will discuss implicationsfor teaching including a comparison of physics majors topre-service elementary teachers followed by the presentationof two individual case studies.

II. BACKGROUND ON PROBLEM SOLVING

RESEARCH

The definition of problem solving that will be used in thispaper comes from Mayer and Whitrock, who say, “Problemsolving is cognitive processing directed at achieving a goalwhen no solution method is obvious to the problem solver.”4

This definition means the designation of a task as a prob-lem must be based on the solver’s response to the task, ratherthan the task itself. If a task does not qualify as a problem fora particular individual, then it becomes merely an exercisefor them. This is an important distinction. Much of the

459 Am. J. Phys. 83 (5), May 2015 http://aapt.org/ajp VC 2015 American Association of Physics Teachers 459

This article is copyrighted as indicated in the article. Reuse of AAPT content is subject to the terms at: http://scitation.aip.org/termsconditions. Downloaded to IP:

72.245.29.161 On: Tue, 02 Jun 2015 23:28:06

classic expert/novice problem solving literature5–7 used tasksthat were exercises for the experts, producing very differentresults than studies that used tasks that were authentic prob-lems for both the experts and novices.8–10 In tackling authen-tic problems, experts did not follow an efficient, working-forward approach as described by the classic expert-noviceliterature. However, experts did spend more time analyzing,planning, and managing their own behavior, demonstrating amore holistic view and systematic problem-solving approach.Their knowledge base also helped them better define theproblem and prevented a wide range of oversights that werecommon to novices.

A. Teaching problem solving

Extensive work has focused on improving students’ prob-lem solving skills in science. Typically problem solving istaught as a whole, utilizing a step-by-step strategy. Theseattempts have met with, at best, limited success. Someresearchers report problem-solving improvement; however,what is meant by improvement in problem solving is notwhat we’ve defined as problem solving above. For example,some see improvement in demonstration of the prescribedsteps, but not in success at finding the correct solutions.11,12

Heller and Hollabaugh13 do find improvement in correct solu-tions of problems when students work in cooperative groups,and Heller and Reif14 show that students perform better,while worked examples are available as references and theyare guided through the solution strategy. However, someresearchers found that teaching problem solving strategies, asrecommended from the above literature, did not result inimproved student performance and that students did not findthese strategies beneficial or worth the extra time.15,16

These step-by-step approaches largely come from thesame research that defines the quintessential expert who effi-ciently solves a problem following an eloquent procedure.However, these procedures come from observing expertssolving exercises, not problems, so they do not representwhat an expert actually does when solving an authentic prob-lem. An additional drawback with students learning to emu-late these eloquent solutions is that they are solutions to atypical textbook problem, where there is a clear problemstatement, a single correct solution, and all informationneeded is given. Such exercises are not representative of theproblems encountered in real-world situations (known as“ill-structured problems” in the problem-solving literature).

B. Characterizing problem-solving sub-skills

Research in cognitive science, psychology, and math oncharacterizing specific types of problem solving skills has typ-ically organized them into two to four categories. Mayer andWhitrock4 categorize skills into knowledge (“knowing”) andprocesses (“doing”). Processes include representing, plan-ning/monitoring, executing, and self-regulating. Knowledgeincludes facts, concepts, strategies, procedures, and beliefs/metacognitive knowledge. Schoenfeld8 framed things slightlydifferently using resources, heuristics, control, and beliefs.Nevertheless, a number of researchers have argued that theseare not adequate, and that what it is that people do when theysolve problems still needs to be better characterized andunderstood.17–19

The PISA 2012 problem-solving framework, built on theresearch discussed above, includes the most complete listing

of sub-skills that we have found in the literature;3 however,it only focuses on processes, and not knowledge or beliefsand motivations. The PISA Problem Solving Group has iden-tified four sets of processes involved in complex problemsolving: exploring and understanding; representing and for-mulating; planning and executing; and monitoring andreflecting. There are 8–15 separate skills listed under each ofthese four processes. Their assessment then ranks the valueof the four processes by theoretical importance and scoresaccording to the value of the skill. But, there is no indicationthat this delineation of skills was empirically tested, and theirassessment only measures a single overall score.

C. problem solving assessment

The state of assessment of specific problem-solving sub-skills has lagged the characterization. Due to the importanceof subject knowledge, the inherent difficulty with distin-guishing the many different sub-skills that a person uses tosolve a complex problem is made worse when one considersassessment of problem solving sub-skills in a particular sub-ject area like physics. If a student lacks a crucial piece ofphysics knowledge, it can be impossible to learn about anyother sub-skills that they have not yet used while solving theproblem. This is the limitation of several problem-solvingassessments developed for use in biology, chemistry, andphysics.20–24 Finally, there are also several assessmentsdesigned to evaluate value-added subject-independent com-petencies in problem solving at the university level. Theseinclude: Collegiate Learning Assessment (CLA) perform-ance task and critique-an-argument task;25 ACT’s CollegiateAssessment of Academic Proficiency (CAAP) Critical think-ing test;26 and ETS’s Measure of Academic Proficiency andProgress (MAPP) Reading and critical thinking.27 Theseassessments only provide a single score to reflect a student’scomposite ability and there is no evidence that what theseassessments measure is relevant to physics problem solving.

III. METHOD

We set out to find an approach that would: empiricallyidentify all the sub-skills that are used in solving a complexproblem; measure the strengths of each of those sub-skills inan individual; and test the relevance of those sub-skills tosolving physics problems. After substantial exploratorywork, we settled on an approach that has a number of novelfeatures.

• We are using a very complex authentic problem, developedby cognitive scientists, to call on a very large number ofsub-skills, but it does not involve any science knowledge.

• We determined that it was easier to initially detect a sub-skill by seeing the impact of its absence rather than itspresence. When a solver is particularly strong in a specificsub-skill, it can be hard to observe its use, so we analyzedand compared a large number of solvers with an extremelywide range of experiences and sub-skills through cognitive“think aloud” interviews. Once we had identified a specificsub-skill by seeing the difficulties encountered by a solverthat lacked that particular sub-skill, we examined indica-tions of the use of that specific sub-skill in the work of thefull range of solvers. This allowed us to identify indica-tions of different levels of that sub-skill displayed by

460 Am. J. Phys., Vol. 83, No. 5, May 2015 W. K. Adams and C. E. Wieman 460

This article is copyrighted as indicated in the article. Reuse of AAPT content is subject to the terms at: http://scitation.aip.org/termsconditions. Downloaded to IP:

72.245.29.161 On: Tue, 02 Jun 2015 23:28:06

solvers, and those indicators were then incorporated intoour problem-solving-assessment rubric.

• We developed a scenario-problem-solving-assessmenttool, where the solver is put in a scripted scenario as theevaluator of two characters who are working through thecomplex problem. This allows the evaluation to continue,even after the solver has revealed critical weaknesses thatwould terminate the solution progress if they were tryingto solve the problem on their own.

The entire methodology was rather lengthy and extendswell beyond physics, and so will be fully presented inanother publication. Here we will present an abbreviatedsummary and the results obtained that are relevant to physicsproblem solving.

A. Identifying sub-skills of problem solving

We used the problem Rescue at Boone’s Meadow fromthe Jasper Woodbury Series, developed by the Cognition andTechnology Group at Vanderbilt University.28 This problemis an everyday complex problem with an embedded datadesign—all content knowledge is provided. The storyinvolves transport of an injured eagle from a remote locationusing a variety of constrained transportation options. Due tothe carefully designed nature of the problem, this assessmentis a true problem for every solver that has been evaluated,from high-school dropouts to physics professors.

A script was created that features two interns talking asthey work through the solution to the problem. A brief intro-duction is provided before the eagle rescue story, explainingto the solver that they are to play the role of a manager in acompany who is responsible for choosing which one of twogood summer interns, Jasmine or Sara, would be offered a

permanent position with the company. The permanent posi-tion would require strong problem solving skills. To assist inthe manager’s decision, the interns would be asked to solve avery involved problem together.

As they work through the script, the solver is presentedwith 40 open-ended questions that are placed at crucialpoints throughout the interns’ discussion. These questionsask the solver’s opinion on how the conversation andproblem-solving process between the interns is progressing.The questions also ask the solver for items of factual knowl-edge, planning, procedures, calculations, and reflections. Theintern script and the questions evolved over time as theywere used in interviews. They started relatively open, but asparticular sub-skills were identified, the script and the ques-tions were modified to more specifically target the informa-tion needed to identify the solver’s level of competence ineach sub-skill.

A set of 18 follow-up questions is placed at the end of theintern script. These questions give the solver a chance to dis-cuss other ideas they may have had that the script did notbring out, probe their ideas about alternative solutions, eval-uate the solver’s metacognitive sub-skills and ask the solverto analyze, in detail, each intern’s problem solving skills.

1. The Sub-skills

Through interviews with the scripted eagle-rescue prob-lem, we found 44 sub-skills that are needed to describe fullythe problem-solving process used by subjects (Table I). Forthe purposes of this research, the definition of a sub-skill is:anything that can affect the subject’s ability to solve theproblem. These sub-skills emerged during the first 20 or sointerviews and no additional sub-skills emerged during 11

Table I. Problem solving sub-skills.

Knowledge—have Beliefs, expectations, and motivation Processes—do

• Math—basic add/sub/mult/div

• Math—equation formation

• Reading comprehension

• Spatial—2-D mapping

• Previously known facts

• Real World knowledge

• Knowledge of own Strengths

• Knowledge of own Weaknesses

• Number Sense

• Estimation

• Ability to analyze others (interns)

• Confidence

• Attribution (takes responsibility for their actions)

• Judgment of information based on the source

• Wants to solve the problem for self

• Wants to solve the problem for interviewer

• Wants to succeed on the “test”

• Interested in the context of the problem

• Enjoyed solving the problem

• Enjoyed analyzing interns

• Enjoyed complete experience

• Real life vs. student

• Careful/Thorough

• Acquires Info 1st time through

• Plan ideas (What—ask questions)

• Plan way to get answer (How)

• Plan—big picture (Visualization)

• Keep problem framework in mind

• Connect steps and pieces

• Check calculations of others

• Aware of how others helped

• Meta-process – step outside of problemsolving to see if own actions are useful.

• Skepticism

• Estimation

• Creativity

• Adaptability (shifts direction easily)

• Can throw out useless info

• Judgment of reasonable issues

• Judgment of importance of numbervalues (is it material)

• Tie in personal experiences

• Tie in info provided by another

• Scientific Process (each step justifiedwith evidence not by gut feeling)

• Remember previously noted facts

• Remember what s/he has calculated or reasoned.

Note: For a description of the specific behavior associated with each category see the problem solving rubric (Ref. 29).

461 Am. J. Phys., Vol. 83, No. 5, May 2015 W. K. Adams and C. E. Wieman 461

This article is copyrighted as indicated in the article. Reuse of AAPT content is subject to the terms at: http://scitation.aip.org/termsconditions. Downloaded to IP:

72.245.29.161 On: Tue, 02 Jun 2015 23:28:06

subsequent interviews. A rubric based on student actions andverbalizations was created to characterize a student’s levelof strength on a five-point scale for each of the sub-skillslisted in Table I.29 The rubric is available online and in thesupplementary material.30

Some of these would not be considered traditionalproblem-solving skills, such as Enjoyed analyzing theinterns; but they fit our definition in that they affected thesubject’s ability to solve the problem. In the earlier stages ofour research, we did not consider these sorts of items to beproblem-solving sub-skills, but we found them necessary tocharacterize how solvers went about solving the problem.However, when we had another researcher evaluate students’problem solving in quantum mechanics using our rubric forthe original list of sub-skills, she asked if we had a way todescribe certain characteristics that she had seen as importantbut were not in the rubric. For example, Wants to find thebest solution for self. We found that all of the characteristicsshe was concerned about were included in our separate listof non-traditional items. We then revised our definition of asub-skill to include all of these elements that describe any-thing that the subject has or does that can affect the subject’ssolution.

2. Categorization

All problem-solving sub-skills are interleaved in a some-what complicated fashion; however, providing a clear frame-work helps one think about the specific instances of failureand provides a language for thinking about and discussingproblem solving. We have divided the list of 44 sub-skillsinto three categories: (1) Knowledge (have); (2) Beliefs,Expectations, and Motivation; and (3) Processes (do). Thefirst and last categories are based on the work of Mayer andWhitrock. However, we found the need for a third category,“Beliefs, Expectations and Motivations,” which mediateswhether a solver decides to apply their knowledge. This cate-gorization also connects the specific types of knowledgeused while engaging with different processes: when formu-lating a mental representation of the problem, solvers use thefacts and concepts that they believe apply to the problem;while planning and monitoring the progress of the plan, solv-ers use problem solving strategies; and, when executing theplan, solvers use the procedural knowledge that they havelearned applies to that specific topic. The interview subjectshave had different combinations of sub-skill strengths andweaknesses, but we saw no indication of any sub-skilldepending on strength in another.

Although we believe this listing of 44 sub-skills is largelycomplete for characterizing all solutions to the eagle rescueproblem, it is not sufficient for all problems. The list of sub-skills under Processes is nearly complete for problems inmany different contexts although more work is needed totest this; the listing of Knowledge sub-skills, however, isinadequate for most specialized problems. This result wasexpected because the instrument was designed to probe howstudents solve problems, rather than to probe the specificknowledge they have. Most evaluations in physics probe theknowledge students have, rather than what they do whensolving problems. For example, in the physics problem solv-ing discussed below, geometry was needed to solve themechanics problem, and many knowledge sub-skills wereneeded to solve the quantum mechanics problems. TheBeliefs, Expectations, and Motivation category is much

closer than the knowledge category to being complete; how-ever, there are likely additional sub-skills of this type thatwill apply to certain problems because a student who is solv-ing a physics (or math, or biology, etc…) problem will bringwith them ideas about what practices are important and/ornecessary for solving a physics problem that would not beapparent in solving the eagle rescue problem.

IV. VALIDATION STUDIES

We carried out validation studies to establish:

(1) That these sub-skills have face validity, in that facultywho are both researchers and teachers see the list of sub-skills as important to solving problems in their disci-plines and are skills they see as valuable for students intheir courses to have.

(2) That people use a similar set of sub-skills with the samelevel of competence when solving physics problems aswhat we measure with the scripted eagle rescue andassociated rubric tool, henceforth referred to as theAssessment of Problem-Solving Skills (APSS).

A. Expert review

Early in the project, interviews were conducted witheleven people who are considered highly successful in theirrespective fields. These included physicists, biophysicists,biochemists, biologists, chemists, mathematicians, account-ants, and business owners. These interviews consisted of ask-ing the interviewee to summarize how they tackle difficultproblems in their field, including different types of problems.The eleven interviews revealed that the skills that were indi-cated as necessary were quite consistent across disciplines(e.g., connect steps and pieces). The only obvious differen-ces were a limited set of skills associated with specific scien-tific knowledge or research processes necessary in thatparticular field. This point came out particularly clearlywhen talking to biochemists or biophysicists, people whocould provide a perspective on solving problems across dif-ferent disciplines.

Later in the project, after identifying the sub-skills listedin Table I, we showed them to 19 university and high-schoolteachers in a range of science disciplines. These teacherswere asked how well the list captured the problem-solvingsub-skills that they felt were important. These teachers allagreed that all of these sub-skills, except enjoyed analyzinginterns, are skills that they value in their discipline and con-sider important for solving problems. They did not identifyany additional sub-skills beyond those on our list.

In an informal test, we also used the APSS to measure thesub-skills of 24 student volunteers (10 were interviewedwhile completing the APSS and 14 provided writtenresponses) from three relatively small physics courses wherethere was extensive instructor-student interaction, and thenasked the instructors for assessments of each student’sstrengths and weaknesses. The instructors were able to pro-vide detailed assessments for every student, and these wereall consistent with what we had measured. However, theAPSS, used with either administration method (1–1.5 h inter-view or written response), provided more detail, and all ofthe instructors indicated that those results explained behav-iors and student difficulties they had seen during the semes-ter but had not understood.

462 Am. J. Phys., Vol. 83, No. 5, May 2015 W. K. Adams and C. E. Wieman 462

This article is copyrighted as indicated in the article. Reuse of AAPT content is subject to the terms at: http://scitation.aip.org/termsconditions. Downloaded to IP:

72.245.29.161 On: Tue, 02 Jun 2015 23:28:06

B. Comparison with solving a mechanics problem

We compared the APSS results with the sub-skills used bystudents solving a mechanics problem during think-aloudinterviews. We chose the Pyramid of Giza problem31 whichrequires the use of Newton’s Laws as well as conservation ofenergy. This problem takes about two hours to solve and isdesigned to require a larger number of sub-skills than typicaltextbook problems.

Five student volunteers were recruited from second termalgebra- and calculus-based introductory physics courses.These students and the interviewer (WA) had not previouslymet. The students were each given the APSS and wrote outanswers to the questions and brought them to theirpyramid-solving interview. An identifier was added to eachcompleted response so that it was anonymous to the grader.The students then participated in think-aloud interviewswhere they worked through the Pyramid of Giza problemand were scored on the problem solving sub-skills theyused.

During the interviews, the interviewer offered to provideany physics content that they requested, but no solutionideas. In all cases, after an hour of working, students wereexhausted and their productivity was visibly reduced. Forthis reason, the interviews were split into one-hour segmentsconducted a week apart. All of the students progressed to apoint where they considered the problem complete, at whichpoint they were asked to reread the question to make surethey had answered everything. They then realized that theyhad not and began working on the next task. At the end ofthe last interview, students were asked follow-up questionsabout their solving process.

The methodology for scoring student’s problem-solvingsub-skills was the same as with the APSS interviews; theinterviewer noted the strength of problem solving sub-skills

as they surfaced during the interviews. Two months after thelast of the pyramid interviews, the APSS written responseswere graded (by WA). Because of the somewhat limited in-formation available with the written responses, between oneand seven of the sub-skills could not be scored for each stu-dent, and the resolution of the scoring was reduced. Inresponse, scoring was condensed to a three-point scale.

As shown in the first two columns of Table II, there werefive sub-skills that did not come out in the pyramid problemthat students did use with the APSS, and there were foursub-skills that were needed for the pyramid problem that arenot used with the APSS.

While the pyramid problem uses more sub-skills than nor-mal textbook problems, it does not require several of theproblem solving sub-skills that are probed with the APSS,such as acquires info the first time through, previously knownfacts, and Adaptability. A notable sub-skill that only showedup on the pyramid problem was what think should knowinhibits effective solving. This sub-skill refers to the diffi-culty students had when they thought they should know aformula or method, whether it existed or not.

After scoring the anonymous APSS responses, it appearedobvious which belonged to which student, and this identifica-tion was confirmed when the written responses were un-blinded. Each student had a pattern of strengths and weak-nesses in solving the pyramid problem, and that samepattern, their “problem-solving fingerprint,” was readilyapparent on the APSS; Fig. 1 shows the comparison for atypical student. Of the 25 sub-skills that could be scoredwith both assessments, 17 had identical scores, and only twohad scores that differed by more than 1. The other studentswere similar. Table III shows the probability of the level ofagreement seen for all students to be �0.001. (See Ref. 32for more detail.)

Table II. Sub-skills that could not be scored with both assessment methods.

Scored with APSS, Not

Pyramid interview

Scored in Pyramid interview,

Not with APSS

Scored in APSS interview,

Not Quantum interview

Scored in Quantum interview,

Not APSS interview

Previously known facts

Ability to analyze interns

Enjoyed Analyzing Interns

Adaptability

Acquires information thefirst time through

Math knowledge-geometry

What think should know inhibitseffective solving

Rounding

Spatial— 3-D visualization

Previously known facts

Ability to analyze interns

Enjoyed Analyzing Interns

Estimation

Interested in the context of theproblem

Number Sense

Spatial— mapping

Careful/Thorough

Judgment of relevance ofnumber values

We did not attempt to score the many Knowledgecomponents involved in solving quantum problemsover the course of a semester.

Fig. 1. Comparison of problem-solving sub-skills as measured by pyramid problem interview (light bars) and the APSS (dark bars). When sub-skills cannot be

scored during an interview, no score is given and appears as a zero in the chart above.

463 Am. J. Phys., Vol. 83, No. 5, May 2015 W. K. Adams and C. E. Wieman 463

This article is copyrighted as indicated in the article. Reuse of AAPT content is subject to the terms at: http://scitation.aip.org/termsconditions. Downloaded to IP:

72.245.29.161 On: Tue, 02 Jun 2015 23:28:06

C. Comparison with solving quantum mechanics

problems

The most rigorous validation study was a comparison ofstudents’ APSS results with their sub-skills measured whilesolving quantum mechanics (QM) problems, where the scor-ing of the APSS and QM assessments were carried out bydifferent people. An independent researcher, S. McKagan,for the purposes of another study,33 had interviewed six QMstudents doing problems every two weeks over the course ofa semester.30 After three days of training with the 5-pointscoring rubric for evaluating sub-skills, McKagan used therubric to score the students sub-skills in solving QM prob-lems by reviewing the videos of her previous QM interviews.It was possible to score the students in 31 of the 40 sub-skillscontained in the scoring rubric at that time (Table II).34

Five of these six students volunteered to participate inAPSS interviews with WA in which their sub-skills weremeasured while solving the eagle-rescue problem. Theresults from the two assessments of these students’ problemsolving sub-skills were then compared. The two assessmentsagreed within 61 point out of 5, on 86% of the 138 itemsscored by both (5 students x 28 sub-skills; not all 31 sub-skills could be scored for every student, with typically 3missing for each.) For 30% of the sub-skills, the two scoreswere identical (Table III).

Each person’s skill set was unique and consistent betweenthe two very different contexts, as illustrated in Figs. 1 and2. This similarity can also be seen in the two interviewer’s

characterizations of individual students, as shown in the fol-lowing example.

Quote from QM interview:“she seemed to view learning [as] how to accept every

weird thing we told her… she thought the first step was toaccept things and the second step was to try to understandthem. She always rethought her ideas when another studentsuggested something, although she maintained enough skep-ticism to recognize that other students were often wrong.”

Quote from eagle rescue interview:“Always had a knee jerk response which was not always

good but then, on her own, she considers carefully andcomes up with the right answer.…she’ll consider whatever isthrown out there, decisions are based on the most logical an-swer. If a suggestion does not make sense after careful con-sideration, she holds onto her beliefs. Very reliable”

This is just an example from one student; however, thesummaries from each student were similarly as consistent,even when there was as much as a year difference betweenwhen the student participated in the quantum mechanicsinterviews and the APSS interview.

These studies showed that all but three of the sub-skillsthat we identified with the APSS were used in solving at leastone of these two physics problems. Two of the three missingsub-skills, previously known facts and ability to analyzeinterns (where one replaces “interns” with “collaborators” or“employees”) would be needed for many authentic real-world problems a physicist might encounter. These resultsalso demonstrate that a person’s level of competency in these

Table III. Consistency of the two sub-skill scores, one obtained from physics problem solving and one from the APSS.

Pyramid vs. APSS Quantum mechanics (QM) vs. APSS

Number of sub-skill scores

for which the two scores

were within D (out of 92).

D¼APSS - Pyramid

scores p-value

Number of sub-skill scores for

which the two scores were

within D (out of 138). D¼APSS - QM scores p-value

49 0 �0.0001 41 0 0.0019

85 61 �0.0001 119 61 �0.0001

… … … 131 62 �0.0001

Note: p-values calculated by finding the chance that scores will randomly agree within D and using the binomial distribution to find the standard deviation.

Fig. 2. Average sub-skill score for the five students (top panel) and for a single student (bottom panel) on the 31 sub-skills as measured by the APSS interviews

(dark bars) and quantum mechanics problem solving interviews (light bars).

464 Am. J. Phys., Vol. 83, No. 5, May 2015 W. K. Adams and C. E. Wieman 464

This article is copyrighted as indicated in the article. Reuse of AAPT content is subject to the terms at: http://scitation.aip.org/termsconditions. Downloaded to IP:

72.245.29.161 On: Tue, 02 Jun 2015 23:28:06

sub-skills transfers between different contexts and problemtypes.

V. IMPLICATIONS FOR TEACHING

We hypothesize that problem solving is readily teachable,but not as a single skill. There are many specific sub-skillsthat must be addressed, one at a time, similar to teachingphysics content knowledge. The work presented here canhelp an instructor recognize and focus activities on individ-ual sub-skills to help students address weaknesses and tobuild up a complete set of problem solving sub-skills. Inaddition, this work will help instructors recognize where dif-ferent students need particular types of help.

A. Physics/engineering students vs pre-service

elementary teachers

Examples of differences between students were clearwhen we compared how physics and engineering majorsdid relative to elementary education majors on the APSS(Table IV).

Physics and engineering majors were able to find solutionsto the problem typically using strong math, planning, andjudgment sub-skills; however, these students did not demon-strate strong knowledge of strengths/weaknesses sub-skills,or engage in meta-processing (stopping and thinking abouttheir current actions). We believe this is because their perso-nal skill set, with which they are proficient, has beenadequate for learning in their major thus far. When facedwith new challenges, missing sub-skill(s) become a problemand metacognitive sub-skills become important to both iden-tify their weaknesses and facilitate the process of correctingweaknesses. Physics majors who were less successful in theircourses tended to be weaker in these same sub-skills than themost successful physics students.

Many elementary-education majors have almost an oppo-site set of strengths and weaknesses from the physics majors.The elementary-education majors who were interviewedhad many more weaknesses. The particular weaknessesvaried but none had a strong, broad skill set resembling aphysics major. Surprisingly, almost all of these elementary-education students had strong metacognitive sub-skills; somewere very careful and thorough when working on a problemand a few even engaged in meta-processing. These sub-skillshelped them compensate for their many other weaknesses inthe knowledge and processes categories.

It is often claimed that elementary-education majors strug-gle with science and/or math because they have low confi-dence in their abilities to do science or math. Out of the eightstudents that were interviewed only two had low confidencein their math/science abilities, although all were, in fact,

weak in math/science (not particularly surprising when con-sidering their lack of experience in math and science).

B. Sub-skills taught in the classroom

These results illustrate a more general educational chal-lenge. Many of the sub-skills listed in Table I are not taught,practiced, or assessed in regular physics courses, and, notsurprisingly, many of those same sub-skills are ones in whichall the students we measured tend to be weak. This holdstrue for both physics majors and elementary-educationmajors, although the courses they take, and the respectivesub-skills they practice in those courses, are quite different.

When thinking about how to teach problem solving, it isuseful to consider that not all sub-skills are required to solveall problems. Back of the chapter textbook problems areshort and usually well defined, so they require only a handfulof the sub-skills that have been identified during this study asbeing needed to solve authentic problems. Many of the sub-skills we have listed go unpracticed and unassessed in typicalphysics classes but are important in the workplace or thelaboratory.

With the awareness that there are different, specific-component problem-solving sub-skills and that certain com-binations are effective for different types of problems,instructors can focus on specific sub-skills by giving studentsappropriate problems that will require the use of thoseskill(s), and provide assessment and feedback targeted tothose sub-skills.

C. Individual student examples

As an example, a physics major at UNC in introductoryphysics (we’ll call him Ben) demonstrated very strong mathand number sense sub-skills and considered a range ofdetails that may affect the solution while solving the eagle-rescue problem during an interview. He remembered all theinformation that he’d acquired and numbers he’d calculated,and kept the problem framework in mind. Ben was also veryverbally critical of the interns. However, he had very weakreading comprehension, was not actually skeptical of the in-formation offered by the interns (he took what they said forgranted and never checked), and was unable to judge reason-able issues for the problem. Ben was aware that he was weakin some areas of problem solving so added extra cushion tohis answers just in case he had made a mistake. The profes-sor’s analysis identified Ben as a weak problem solver but avery conscientious student with strong math sub-skills. Overtime the professor had discovered that Ben appeared skepti-cal in group environments, yet never thought through whatwas offered by others, and so was often being led astray.The one feature the professor had not identified was Ben’sweak reading ability. The professor commented that this

Table IV. Physics majors vs. Elementary Ed majors relative strengths and weaknesses.

Common strengths Common weakness

Physics/engineering

students

Math— equation formation, Math— number sense,

planning, creativity, and judgment

Knowledge of strengths/weaknesses, Meta-processing

(stopping and thinking about their current actions),

and Careful and Thorough

Pre-service elementary

teachers

Knowledge of strengths/weaknesses,

and Careful and thorough

Creativity Varied but many and serious (i.e., Planning big picture,

what and how, reading comprehension, acquires information1st time through)

465 Am. J. Phys., Vol. 83, No. 5, May 2015 W. K. Adams and C. E. Wieman 465

This article is copyrighted as indicated in the article. Reuse of AAPT content is subject to the terms at: http://scitation.aip.org/termsconditions. Downloaded to IP:

72.245.29.161 On: Tue, 02 Jun 2015 23:28:06

knowledge could have helped him do a better job of address-ing Ben’s problems in class. Most of what the professor sur-mised about Ben took the better part of a semester of carefulobservation to identify. If he’d known at the beginning of thesemester where Ben was weak, he could have addressedthese issues early on.

Another example of the potential value to instructors ofrecognizing this detailed set of problem-solving sub-skills isprovided by a student we will call Janet. In class, Janet didquite well in group activities, especially those involving cal-culations and/or working through data. She was a muchsought-after group member. When interacting with hergroup, it was clear that Janet always knew how they’darrived at their solution while the other students often hadmissed some aspect. However, when working on an individ-ual project involving the synthesis of ideas from two sources,Janet struggled. This seemed quite surprising and hard tounderstand considering her overall class performance. At theend of the semester, Janet completed the eagle-rescue assess-ment interview and this made it clear that Janet was extraor-dinarily weak at formulating questions on her own. After30 min of making no progress on the first question in thescript, which required her to indicate what the interns neededto figure out, the interviewer encouraged her to move on tosee what the interns would do. She then went through thescript answering the questions easily. If the interns had anyideas about what might be useful, Janet was able to figureout how to find that information and carry out whatever cal-culation was needed. Throughout the interview, however,she demonstrated an inability to ask her own questions—what might be needed for the next step in the solution—buthad no difficulty in finding the answers to any question posedand evaluating its usefulness. Janet graduated with straightA’s in spite of being unable to make progress on any prob-lem that did not have a clearly articulated question. Thisfairly serious weakness was never addressed, because shewas in a major that is taught almost 100% in group format,so she was able to rely on group members to formulate ques-tions, and normal course exams involve well-articulatedquestions.

D. Discussion

To learn any skill a person must practice it. Here, we havedescribed which sub-skills are commonly practiced by stu-dents doing textbook problems and identified those sub-skillsthat are rarely needed to be successful in the typical under-graduate physics classroom. Being aware of this allows aphysics instructor (teaching any level of physics) the oppor-tunity to consider where s/he may or may not incorporatepractice of these other sub-skills. Whether one considersthese sub-skills as program-level or course-level goals, it isimportant to identify where students will learn these sub-skills if they are part of the desired outcome.

We often have colleagues ask us if there are specific skillsthat are more general or more important than others.Different problems require different sub-skills to solve them.Similarly, different sub-skills have varying levels of impor-tance for successful solution of different problems. As notedabove we did find that different sets of sub-skills defined astrong physics major compared to a strong pre-service ele-mentary teacher. The relative importance of sub-skills isdefined by the particular problem, similar to the definition ofa problem being defined by the solver and not just the task.

It would be interesting to compare skills and processes inthe problem-solving literature to our sub-skills and catego-ries. Here, we offer a couple of examples. In most cases, theskills described in the literature subsume many sub-skills.One example is “heuristics,” which would fit within our cate-gory of knowledge that students have. Heuristics are “rulesof thumb” or strategies such as exploiting analogies or work-ing backward and are something students learn to use tosolve certain categories of problems making them domainspecific. As mentioned earlier, the knowledge category isbasically infinite since it contains content knowledge. Asecond example would be the skills identified in Docktorand Heller’s rubric.21 Three of these skills, “physicsapproach, specific application of physics, and mathematicalprocedures,” are knowledge, measures of specific contentsub-skills needed to solve each problem while “logical pro-gression” would closely map to scientific process (each stepjustified with evidence not by gut feeling).

VI. CONCLUSION

We have empirically identified 44 distinct sub-skills thataffect a person’s ability to solve problems in many differentcontexts. We see that a person brings the same problem-solving fingerprint representing their strengths and weak-nesses across these sub-skills to problems they encounter indifferent contexts. In particular, we see students displayingnearly the same sub-skill fingerprint when solving both amechanics problem and introductory quantum mechanicsproblems as they displayed when solving a non-physicsproblem. The existence of so many distinct important sub-skills explains why it has been so difficult to teach or assessproblem solving as a single distinct skill, and this work willallow physics teachers to better target their problem-solvinginstruction. A goal of future work is to create assessmenttools that will not require as much time and training to useand will provide reliable measures of at least many, if notall, of these sub-skills used in physics problem solving.

ACKNOWLEDGMENTS

The authors are pleased to acknowledge the invaluablehelp of Sarah McKagan as well as the contributions of all ofthe interview volunteers and the faculty members whoprovided assistance. The authors also acknowledge helpfulcomments from an anonymous reviewer on clarifying thediscussion. This work was funded by the NSF, theUniversity of Colorado at Boulder, and the University ofNorthern Colorado.

1D. P. Maloney, “An Overview of Physics Education Research on Problem

Solving,” in Getting Started in PER, edited by C. Henderson and K. A.

Harper (American Association of Physics Teachers, College Park, MD,

2011), Reviews in PER Vol. 2, <http://www.per-central.org/items/

detail.cfm?ID¼11457>.2National Research Council (NRC), Adapting to a Changing World—Challenges and opportunities in Undergraduate Physics Education(National Academies Press, Washington, DC, 2013).

3OECD, PISA 2012 Assessment and Analytical Framework: Mathematics,Reading, Science, Problem Solving and Financial Literacy, <http://

dx.doi.org/10.1787/9789264190511-en> (2013).4R. E. Mayer and M. Whitrock, “Problem Solving,” in Handbook ofEducational Psychology, 2nd ed., edited by Patricia A. Alexander and

Philip H. Winne (Erlbaum, New Jersey, 2006), Chap. 13, pp. 287–303.

466 Am. J. Phys., Vol. 83, No. 5, May 2015 W. K. Adams and C. E. Wieman 466

This article is copyrighted as indicated in the article. Reuse of AAPT content is subject to the terms at: http://scitation.aip.org/termsconditions. Downloaded to IP:

72.245.29.161 On: Tue, 02 Jun 2015 23:28:06

5J. H. Larkin, “Processing Information for Effective Problem Solving,”

Eng. Educ. 70(3), 285–288 (1979).6J. H. Larkin, J. McDermott, D. P. Simon, and H. A. Simon, “Expert and

novice performance in solving physics problems,” Science 208(4450),

1335–1342 (1980).7M. T. H. Chi, P. S. Feltovich, and R. Glaser, “Categorization and represen-

tation of physics problems by experts and novices,” Cogn. Sci. 5, 121–152

(1981).8A. H. Schoenfeld, Mathematical Problem Solving (Academic Press, Inc.,

Harcourt Brace Javanovich, Orlando, 1985).9C. Singh, “When physical intuition fails,” Am. J. Phys. 70, 1103–1109

(2002).10S. Wineburg, “Reading Abraham Lincoln: An expert/expert study in the

interpretation of historical texts,” Cogn. Sci. 22, 319–346 (1998).11P. Heller, R. Keith, and S. Anderson, “Teaching problem solving through

cooperative grouping. Part 1: Group versus individual problem solving,”

Am. J. Phys. 60(7), 627–636 (1992).12D. Huffman, “Effect of explicit problem solving instruction on high school

students’ problem-solving performance and conceptual understanding of

physics,” J. Res. Sci. Teach. 34(6), 551–570 (1997).13P. Heller and M. Hollabaugh, “Teaching problem solving through coopera-

tive grouping. Part 2: Designing problems and structuring groups,” Am. J.

Phys. 60(7), 637–644 (1992).14J. I. Heller and F. Reif, “Prescribing effective human problem-solving

processes: Problem description in physics,” Cogn. Instrum. 1(2), 177–216

(1984).15D. M. Bunce and H. Heikkinen, “The effects of an explicit problem-

solving approach on mathematical chemistry achievement,” J. Res. Sci.

Teach. 23, 11–20 (1986).16G. DiLisi, J. Eulberg, J. Lanese, and P. Padovan, “Establishing problem-

solving habits in introductory science courses,” J. College Sci. Teach.

35(5), 42–47 (2006).17D. P. Maloney, “Research on Problem Solving: Physics,” in Handbook of

Research on Science Teaching and Learning Project, edited by D. L.

Gabel (MacMillan Publishing Company, New York, 1993).18J. R. Anderson and C. D. Schunn, “Implications of the ACT-R learning

theory: No magic bullets,” in Advances in instructional psychology, edited

by R. Glaser (Lawrence Erlbaum Associates, New Jersey, 2000), Vol. V,

pp. 1–34.19B. Ross, “Cognitive science: Problem solving and learning for physics

education,” AIP Conf. Proc. 951, 11–14 (2007).

20W. L. Anderson, C. A. Sensibaugh, M. P. Osgood, and S. M. Mitchell,

“What really matters: Assessing individual problem-solving performance

in the context of biological sciences,” Int. J. Schol. Teach. Learn. 5(1), 17-

1–20 (2011).21J. Docktor and K. Heller, “Robust assessment instrument for student prob-

lem solving,” in Proc. NARST 2009 Annual (NARST, Reston, VA, 2009).22E. E. Gayon, The Problem-solving ability of high school chemistry

students and its implications in redefining chemistry education, <http://

academia.edu/593365/THE_PROBLEM-SOLVING_ABILITY_OF_HIGH_

SCHOOL_CHEMISTRY_STUDENTS_AND_ITS_IMPLICATIONS_IN_

REDEFINING_CHEMISTRY_EDUCATION> (2005).23D. Hestenes and M. Wells, “A mechanics baseline test,” Phys. Teach. 30,

159–166 (1992).24J. Marx and K. Cummings, “Development of a survey instrument to gauge

students’ problem-solving abilities,” AIP Conf. Proc. 1289, 221–224

(2010).25S. Klein, R. Benjamin, R. Shavelson, and R. Bolus, “The collegiate learn-

ing assessment: Facts and fantasies,” Eval. Rev. 31, 415–439 (2007).26ACT CAAP Technical Handbook (ACT, Iowa City, IA, 2007).27Educational Testing Service (ETS), MAPP User’s Guide (Educational

Testing Service, Princeton, NJ, 2007).28Cognition and Technology Group at Vanderbilt, The Jasper Project:

Lessons in Curriculum, Instruction, Assessment, and ProfessionalDevelopment (Lawrence Erlbaum Associates, Mahwah, New Jersey, 1997).

29W. K. Adams, Grading Rubric, <http://www.unco.edu/nhs/physics/fac-

ulty/adams/Research/Rubric.pdf> (2007).30See supplementary material at http://dx.doi.org/10.1119/1.4913923 for

problem solving rubric; the Great Pyramid of Piza problem; McKagen’s

interview protocol and an example of a quantum mechanics problem used

in these interviews.31University of Maryland Physics Education Group, Alternative homework

Assignments (AHA), <http://www.physics.umd.edu/perg/abp/aha/index.

html> (2002).32W. K. Adams, Development of a Problem Solving Evaluation Instrument;

Untangling of specific problem solving skills, <www.unco.edu/nhs/

physics/faculty/adams/research.html> (2008).33S. B. McKagan, K. K. Perkins, and C. E. Wieman, “Reforming a large lec-

ture modern physics course for engineering majors using a PER-based

design,” AIP Conf. Proc. 883, 34–37 (2007).34W. K. Adams and C. E. Wieman, “Problem solving skill evaluation instru-

ment—validation studies,” AIP Conf. Proc. 883, 18–21 (2007).

467 Am. J. Phys., Vol. 83, No. 5, May 2015 W. K. Adams and C. E. Wieman 467

This article is copyrighted as indicated in the article. Reuse of AAPT content is subject to the terms at: http://scitation.aip.org/termsconditions. Downloaded to IP:

72.245.29.161 On: Tue, 02 Jun 2015 23:28:06