What is mathematical understanding? How can we measure it?

Theoretical and empirical reflections

Dr. Jon STAR

Harvard University

Seminar Presentation, Department of Elementary Education Seminar, Middle East Technical University, Ankara, Turkey; 27 August 2007

2

Acknowledgements

• Dr. Cigdem HASER– Department of Elementary Education

• METU

3

Some background information

• Who is Dr. Jon STAR?• What is his connection to METU?• What is the format of this seminar?• Who is the intended audience for this seminar?• Can I interrupt the seminar with questions?

4

My goals for the seminar

• Push you to think more carefully about – What “understanding” means– How understanding is assessed

• Convince you (if you need convincing!) that the nature of and assessment of understanding are issues that you need to think seriously about in your work with teachers and students

5

Getting started

• Please get in groups of 2-3 people and share your answers to the following questions:

1. What does it mean to understand a mathematical topic?

2. How can you tell when someone has developed understanding of a mathematical topic?

6

“Understanding” words?

• understand, know, explain, concept, procedure, rote, memorize, transfer, apply, recall, generate, prove, problem solving, network, connections, web, schema, ...

7

Does this assess understanding?

8

Does this assess understanding?

9

Does this assess understanding?

10

Does this assess understanding?

11

Does this assess understanding?

12

Does this assess understanding?

13

Generalizing?

• Can you make any general statements about what understanding is and how to assess it?

• Five categories of responses to the question– “What does it mean for someone to understand a

mathematical topic”?”

• And my thoughts on each...

14

1. Understanding is hard to define.

• I know it when I see it, but I can’t define it• Understanding is difficult to define precisely• People smarter than I am have tried and failed

over the past several thousand years!• Is it really necessary to come up with a good

definition of “understanding”?

What is understanding?

15

Jon says...

• YES, it is necessary to come up with a definition• Understanding is often the instructional outcome

that we aim for. – We need a target when we are designing instructional

materials

• Need to measure whether our goal has been achieved– Designing measures requires a careful and precise

definition and operationalization of understanding

1. Understanding is hard to define.

16

2. Cite an expert.

• You can’t expect me to re-invent the wheel!• I use understanding in the same way that (insert

your favorite mathematics education scholar here) does.– For example, Hiebert, Skemp, Brownell, Sfard, Star?

• If another framework is widely used and accepted, isn’t it OK to just use it?

What is understanding?

17

Jon says...

• Certainly it is important to ground your ideas about understanding with the established literature

• It helps to connect with a literature that is well-known and a framework for understanding that is commonly used

• But...

2. Cite an expert.

18

Jon says...

• Most who “piggy-back” on others’ frameworks are not particularly knowledgeable about who and what they are citing– Pervasive citation of Hiebert (1986) and Skemp

(1978) fall into this category

• Before citing someone’s ideas about understanding, become very familiar with the framework

2. Cite an expert.

19

Jon says...

• Many assume that different frameworks mean the same thing– For example, that procedural/conceptual (Hiebert) is

the same as relational/instrumental (Skemp)

• Different frameworks and terminological distinctions shed light on different aspects of understanding– All theoretical frameworks are not the same

2. Cite an expert.

20

Assessment frameworks

• Many adopt a framework because of its use in national or international assessments– Widespread use of an assessment framework is not a

compelling reason to choose a framework

• Assessment frameworks are idiosyncratic and difficult to faithfully apply outside of the assessment– Assessment frameworks often not theoretically sound

2. Cite an expert.

21

Consider these frameworks

• NAEP (US national maths assessment)– Procedural knowledge, Conceptual understanding,

Problem solving

• PISA (large international maths assessment)– Reproduction, Connections, Reflection

• TIMSS (large international maths assessment)– Solving routine problems, Knowing facts and

procedures, Reasoning, Using concepts

2. Cite an expert.

22

TIMSS 2003 8th grade M012042

Knowing facts and proceduresSolving routine problemsKnowing facts and proceduresReasoningUsing concepts

23

TIMSS 2003 8th grade M022002

Solving routine problemsSolving routine problemsKnowing facts and proceduresReasoningUsing concepts

24

TIMSS 2003 8th grade M022251

Using conceptsSolving routine problemsKnowing facts and proceduresReasoningUsing concepts

25

TIMSS 2003 8th grade M022008

Reasoning

Solving routine problemsKnowing facts and proceduresReasoningUsing concepts

26

NAEP 8th grade 2003-8M6 #27

Conceptual understandingConceptual understandingProcedural knowledgeProblem solving

27

NAEP 8th grade 2003-8M7 #18

Procedural knowledgeConceptual understandingProcedural knowledgeProblem solving

28

NAEP 8th grade 2003-8M7 #11

Problem solvingConceptual understandingProcedural knowledgeProblem solving

29

Jon says...

• Know what you are getting into when you cite an expert!

• Don’t assume that assessment frameworks reflect a coherent and theoretically sound vision of what understanding is

2. Cite an expert.

30

3. Knowledge organization in head.

• Understanding refers to how knowledge is organized in someone’s memory/brain/head.

• Understanding is when knowledge is tightly connected, is a web, is a network, is organized into schema, etc.

What is understanding?

31

Jon says...

• It may be true that understanding is indicated by or a result of a certain organization of knowledge in the head

• It may be helpful to use metaphors and abstractions (such as links and connections) to talk about knowledge organization

• But...

3. Knowledge organization in head.

32

Jon says...

• Defining understanding this way is not particularly helpful in designing assessments

• The metaphors or abstractions are not directly transferable to learning and assessment– If links = understanding, do we teach links?– Does the ability to draw a concept map (with lots of

links in a network) indicate understanding?

3. Knowledge organization in head.

33

4. Quality of verbalization.

• A student understands if he/she can explain it to someone else.

• If the student knows it well enough to provide a thorough and principled explanation to someone else, I am satisfied that he/she understands.

What is understanding?

34

Jon says...

• Certainly the ability to verbally explain is a potential indicator that a student understands

• But...

4. Quality of verbalization.

35

Jon says...

• Impractical in terms of assessment; we can’t interview every child to see if he/she understands– We need a more efficient way to tap understanding

• Students’ verbalizations are often spontaneous– Typically not a well-thought-out, carefully worded,

articulate response– Requires an interviewer to carefully probe and explore

the student’s ZPD

4. Quality of verbalization.

36

5. Quality of written performance.

• Understanding is evident in what students do• When given a task/problem, we judge whether

or not the student understands based on his/her performance

• Verbalization can give an additional window into what students are thinking, and knowledge organization is potentially useful as a metaphor

• “Intelligent performances” (Ryle)

What is understanding?

37

Jon says...

• Closest to what I believe• Major implications for assessment

– It is incumbent on me to design very good questions so that how students respond does indicate that they understand

• Let’s talk about creating good assessments of understanding!

5. Quality of written performance.

38

Assessments of understanding

• Don’t be swayed by existing frameworks for categorizing items or by the assignment of particular items to categories

• Questions from other assessments may be quite useful, but the category label (“Solving routine problems”) is likely not helpful

39

Avoid stereotypes

• Stereotyped views of conceptual knowledge– “If it is possible to complete this question using a

memorized procedure, then it does not assess understanding”

– “If it is a multiple choice question, it cannot assess conceptual understanding”

40

Avoid stereotypes

• Stereotyped views of procedural knowledge– Procedures are either known (and student can

execute them, often by rote) or they are not known

• Also consider items that assess “deep procedural knowledge”– knowledge of multiple procedures, knowledge of which

procedures are better for certain circumstances, ability to adapt procedures to changing circumstances, ability to evaluate procedures

See Star, 2005, 2007, articles in J. for Research in Mathematics Education for more on this.

41

Avoid “problem solving”

• Many assessment frameworks include a category called “problem solving”

• Overlaps with other categories (e.g., procedural knowledge and conceptual knowledge)

• Word problems fall into multiple categories, not just problem solving

• Has become a political term and thus relatively useless for mathematics educators

42

Name the concept

• When designing assessments of conceptual knowledge, it is important to be able to name the target concept

• Conceptual knowledge as knowledge of a concept– Rather than as a descriptor for a quality of knowledge

43

Validity

• Evaluate the psychometic validity of your assessments– Cronbach’s alpha for inter-item reliability– Factor analyses to establish or confirm ‘grouping’ of

items

44

Bottom line

• Be thoughtful and deliberate about the frameworks, terms, and citations that you use

• Make sure your definition of understanding is tightly linked to the ways that you operationalize and assess it

45

Examples from my research

• I assess three components of understanding (of algebra equation solving)

• Conceptual knowledge• Procedural knowledge• Procedural flexibility

46

Conceptual knowledge

• Knowledge of key concepts used in equation solving– Concept of equivalence– Concept of variable

47

For example

• If m is a positive number, which of these is equivalent to (the same as) m + m + m + m? (Responses are: 4m; m4; 4(m + 1); m + 4)– Concept of variable

• For the two equations:213x + 476 = 984 213x + 476 + 4 = 984 + 4 Without solving either equation, what can you say about the answers to these equations? Explain your answer.– Concept of equivalence

48

Procedural knowledge

• Knowledge of procedures– Ability to successfully execute equation solving

procedures on problems that are similar to those seen in the instructional intervention (“familiar” or “learning” problems)

– Ability to successfully execute equation solving procedures that are somewhat different than those seen in the intervention (“transfer” problems)

49

Procedural flexibility• Ability to generate, recognize, and evaluate

multiple solution methods for the same problem

• One example of “deep procedural knowledge”• Short-answer and multiple choice assessment

designed to measure flexibility

(e.g., Beishuizen, van Putten, & van Mulken, 1997; Blöte, Klein, & Beishuizen, 2000; Blöte, Van der Burg, & Klein, 2001; Star, 2005, 2007; Star & Seifert, 2006; Rittle-Johnson & Star, 2007)

50

Examples of flexibility items

• A student’s first step for solving the equation 3(x + 2) = 12 was x + 2 = 4. What step did the student use to get from the first step to the second step? Do you think that this way of starting this problem is (a) a very good way; (b) OK to do, but not a very good way; (c) Not OK to do? Explain your reasoning.

• For the equation 2(x + 1) + 4 = 12, identify all possible steps (among 4 given choices) that could be done next.

• Solve 4(x + 2) = 12 in two different ways.

51

Also, direct measure of flexibility

• Look at students’ strategies on procedural knowledge items

• How they solve equations, and what this indicates about their knowledge of equation solving strategies

52

Successes

• Reliable measure of flexibility that does assess an aspect of understanding that is of particular interest to me– Creating and refining the assessment also forced us

to think deeply about what flexibility means

• Thinking about understanding and thinking about assessing understanding interact with each other

53

Challenges

• Creating a valid assessment for conceptual knowledge– Several items that seem to target key concepts– Items are successful in measuring students’ learning

from our instructional intervention– But items as a whole are not as strong from a

psychometric standpoint as we would like

Thanks!

You can download this presentation and other related papers and talks at

http://gseacademic.harvard.edu/~starjo

Jon StarJon_Star@Harvard.edu